view docs/user.tex @ 2570:30a9b33481dc

bitkeeper revision 1.1159.90.2 (415be9d5hTw1zLV9fA-AYcekmwhMwg)

Discard devices early for local migrate.
author mjw@wray-m-3.hpl.hp.com
date Thu Sep 30 11:11:17 2004 +0000 (2004-09-30)
parents 11be1dfb262b
children b3ae9d6dbece
line source
1 \documentclass[11pt,twoside,final,openright]{xenstyle}
2 \usepackage{a4,graphicx,setspace}
3 \setstretch{1.15}
4 \input{style.tex}
6 \begin{document}
9 \pagestyle{empty}
10 \begin{center}
11 \vspace*{\fill}
12 \includegraphics{eps/xenlogo.eps}
13 \vfill
14 \vfill
15 \vfill
16 \begin{tabular}{l}
17 {\Huge \bf Users' manual} \\[4mm]
18 {\huge Xen v2.0 for x86} \\[80mm]
20 {\Large Xen is Copyright (c) 2004, The Xen Team} \\[3mm]
21 {\Large University of Cambridge, UK} \\[20mm]
22 {\large Last updated on 30th July, 2004}
23 \end{tabular}
24 \vfill
25 \end{center}
26 \cleardoublepage
29 \pagestyle{plain}
30 \pagenumbering{roman}
31 { \parskip 0pt plus 1pt
32 \tableofcontents }
33 \cleardoublepage
36 \pagenumbering{arabic}
37 \raggedbottom
38 \widowpenalty=10000
39 \clubpenalty=10000
40 \parindent=0pt
41 \renewcommand{\topfraction}{.8}
42 \renewcommand{\bottomfraction}{.8}
43 \renewcommand{\textfraction}{.2}
44 \renewcommand{\floatpagefraction}{.8}
45 \setstretch{1.15}
47 \newcommand{\path}[1]{{\tt #1}}
49 \part{Introduction and Tutorial}
50 \chapter{Introduction}
52 {\bf
53 DISCLAIMER: This documentation is currently under active development
54 and as such there may be mistakes and omissions --- watch out for
55 these and please report any you find to the developer's mailing list.
56 Contributions of material, suggestions and corrections are welcome.
57 }
59 Xen is a { \em paravirtualising } virtual machine monitor (VMM) or
60 ``Hypervisor'' for the x86 processor architecture. Xen can securely
61 multiplex heterogeneous virtual machines on a single physical with
62 near-native performance. The virtual machine technology facilitates
63 enterprise-grade functionality, including:
65 \begin{itemize}
66 \item Virtual machines with close to native performance.
67 \item Live migration of running virtual machines.
68 \item Excellent hardware support (use unmodified Linux device drivers).
69 \item Suspend to disk / resume from disk of running virtual machines.
70 \item Transparent copy on write disks.
71 \item Sandboxed, restartable device drivers.
72 \item Pervasive debugging - debug whole OSes, from kernel to applications.
73 \end{itemize}
75 Xen support is available for increasingly many operating systems. The
76 following OSs have either been ported already or a port is in
77 progress:
78 \begin{itemize}
79 \item Dragonfly BSD
80 \item FreeBSD 5.3
81 \item Linux 2.4
82 \item Linux 2.6
83 \item NetBSD 2.0
84 \item Plan 9
85 \item Windows XP
86 \end{itemize}
88 Right now, Linux 2.4 and 2.6 are available for Xen 2.0. NetBSD
89 port will be updated to run on Xen 2.0, hopefully in time for the NetBSD
90 2.0 release. It is intended that Xen support be integrated into the
91 official releases of Linux 2.6, NetBSD 2.0, FreeBSD and Dragonfly BSD.
93 Even running multiple copies of Linux can be very useful, providing a
94 means of containing faults to one OS image, providing performance
95 isolation between the various OS instances and trying out multiple
96 distros.
98 The Windows XP port is only available to those who have signed the
99 Microsoft Academic Source License.
101 Possible usage scenarios for Xen include:
102 \begin{description}
103 \item [Kernel development] test and debug kernel modifications in a
104 sandboxed virtual machine --- no need for a separate test
105 machine
106 \item [Multiple OS Configurations] run multiple operating systems
107 simultaneously, for instance for compatibility or QA purposes
108 \item [Server consolidation] move multiple servers onto one box,
109 provided performance and fault isolation at virtual machine
110 boundaries
111 \item [Cluster computing] improve manageability and efficiency by
112 running services in virtual machines, isolated from
113 machine-specifics and load balance using live migration
114 \item [High availability computing] run device drivers in sandboxed
115 domains for increased robustness
116 \item [Hardware support for custom OSes] export drivers from a
117 mainstream OS (e.g. Linux) with good hardware support
118 to your custom OS, avoiding the need for you to port existing
119 drivers to achieve good hardware support
120 \end{description}
122 \section{Structure}
124 \subsection{High level}
126 A Xen system has multiple layers. The lowest layer is Xen itself ---
127 the most privileged piece of code in the system. On top of Xen run
128 guest operating system kernels. These are scheduled pre-emptively by
129 Xen. On top of these run the applications of the guest OSs. Guest
130 OSs are responsible for scheduling their own applications within the
131 time allotted to them by Xen.
133 One of the domains --- { \em Domain 0 } --- is privileged. It is
134 started by Xen at system boot and is responsible for initialising and
135 managing the whole machine. Domain 0 builds other domains and manages
136 their virtual devices. It also performs suspend, resume and
137 migration of other virtual machines. Where it is used, the X server
138 is also run in domain 0.
140 Within Domain 0, a process called ``Xend'' runs to manage the system.
141 Xend is responsible for managing virtual machines and providing access
142 to their consoles. Commands are issued to Xend over an HTTP
143 interface, either from a command-line tool or from a web browser.
145 XXX need diagram(s) here to make this make sense
147 \subsection{Paravirtualisation}
149 Paravirtualisation allows very high performance virtual machine
150 technology, even on architectures (like x86) which are traditionally
151 hard to virtualise.
153 Paravirtualisation requires guest operating systems to be { \em ported
154 } to run on the VMM. This process is similar to a port of an
155 operating system to a new hardware platform. Although operating
156 system kernels must explicitly support Xen in order to run in a
157 virtual machine, { \em user space applications and libraries
158 do not require modification }.
160 \section{Hardware Support}
162 Xen currently runs on the x86 architecture, but could in principle be
163 ported to others. In fact, it would have been rather easier to write
164 Xen for pretty much any other architecture as x86 is particularly
165 tricky to handle. A good description of Xen's design, implementation
166 and performance is contained in the October 2003 SOSP paper, available
167 at:\\
168 {\tt http://www.cl.cam.ac.uk/netos/papers/2003-xensosp.pdf}\\
169 Work to port Xen to x86\_64 and IA64 is currently underway.
171 Xen is targeted at server-class machines, and the current list of
172 supported hardware very much reflects this, avoiding the need for us
173 to write drivers for "legacy" hardware. It is likely that some desktop
174 chipsets will fail to work properly with the default Xen
175 configuration: specifying {\tt noacpi} or {\tt ignorebiostables} when
176 booting Xen may help in these cases.
178 Xen requires a ``P6'' or newer processor (e.g. Pentium Pro, Celeron,
179 Pentium II, Pentium III, Pentium IV, Xeon, AMD Athlon, AMD Duron).
180 Multiprocessor machines are supported, and we also have basic support
181 for HyperThreading (SMT), although this remains a topic for ongoing
182 research. We're also working on an x86\_64 port (though Xen should
183 already run on these systems just fine in 32-bit mode).
185 Xen can currently use up to 4GB of memory. It is possible for x86
186 machines to address up to 64GB of physical memory but (unless an
187 external developer volunteers) there are no plans to support these
188 systems. The x86\_64 port is the planned route to supporting more
189 than 4GB of memory.
191 In contrast to previous Xen versions, in Xen 2.0 device drivers run
192 within a privileged guest OS rather than within Xen itself. This means
193 that we should be compatible with the majority of device hardware
194 supported by Linux. The default XenLinux build contains support for
195 relatively modern server-class network and disk hardware, but you can
196 add support for other hardware by configuring your XenLinux kernel in
197 the normal way (e.g. \verb_# make ARCH=xen xconfig_).
199 \section{History}
202 ``Xen'' is a Virtual Machine Monitor (VMM) originally developed by the
203 Systems Research Group of the University of Cambridge Computer
204 Laboratory, as part of the UK-EPSRC funded XenoServers project.
206 The XenoServers project aims to provide a ``public infrastructure for
207 global distributed computing'', and Xen plays a key part in that,
208 allowing us to efficiently partition a single machine to enable
209 multiple independent clients to run their operating systems and
210 applications in an environment providing protection, resource
211 isolation and accounting. The project web page contains further
212 information along with pointers to papers and technical reports:
213 {\tt http://www.cl.cam.ac.uk/xeno}
215 Xen has since grown into a project in its own right, enabling us to
216 investigate interesting research issues regarding the best techniques
217 for virtualizing resources such as the CPU, memory, disk and network.
218 The project has been bolstered by support from Intel Research
219 Cambridge, and HP Labs, who are now working closely with us. We're
220 also in receipt of support from Microsoft Research Cambridge to port
221 Windows XP to run on Xen.
223 Xen was first described in the 2003 paper at SOSP \\
224 ({\tt http://www.cl.cam.ac.uk/netos/papers/2003-xensosp.pdf}).
225 The first public release of Xen (1.0) was made in October 2003. Xen
226 was developed as a research project by the University of Cambridge
227 Computer Laboratory (UK). Xen was the first Virtual Machine Monitor
228 to make use of {\em paravirtualisation} to achieve near-native
229 performance virtualisation of commodity operating systems. Since
230 then, Xen has been extensively developed and is now used in production
231 scenarios on multiple sites.
233 Xen 2.0 is the latest release, featuring greatly enhanced hardware
234 support, configuration flexibility, usability and a larger complement
235 of supported operating systems. We think that Xen has the potential
236 to become {\em the} definitive open source virtualisation solution and
237 will work to conclusively achieve that position.
240 \chapter{Installation}
242 The Xen distribution includes three main components: Xen itself,
243 utilities to convert a standard Linux tree to run on Xen and the
244 userspace tools required to operate a Xen-based system.
246 This manual describes how to install the Xen 2.0 distribution from
247 source. Alternatively, there may be packages available for your
248 operating system distribution.
250 \section{Prerequisites}
251 \label{sec:prerequisites}
252 \begin{itemize}
253 \item A working installation of your favourite Linux distribution.
254 \item A working installation of the GRUB bootloader.
255 \item An installation of Twisted v1.3 or above (see {\tt
256 http://www.twistedmatrix.com}). There may be a package available for
257 your distribution; alternatively it can be installed by running {\tt \#
258 make install-twisted} in the root of the Xen source tree.
259 \item The Linux bridge control tools (see {\tt
260 http://bridge.sourceforge.net}). There may be a packages of these
261 tools available for your distribution.
262 \item Linux IP Routing Tools
263 \item make
264 \item python-dev
265 \item gcc
266 \item zlib-dev
267 \item libcurl
268 \item python2.3-pycurl
269 \item python2.3-twisted
270 \end{itemize}
272 \section{Optional}
273 \begin{itemize}
274 \item The Python logging package (see {\tt http://www.red-dove.com/})
275 for additional Xend logging functionality.
276 \end{itemize}
278 \section{Install Bitkeeper (Optional)}
280 To fetch a local copy, first download the BitKeeper tools at: \\ {\tt
281 http://www.bitmover.com/download }
283 The BitKeeper install program is designed to be run with X. If X is
284 not available, you can specify the install directory on the command
285 line.
287 \section{Download the Xen source code}
289 \subsection{Using Bitkeeper}
291 The public master BK repository for the 2.0 release lives at: \\
292 {\tt bk://xen.bkbits.net/xen-2.0.bk}. You can use Bitkeeper to
293 download it and keep it updated with the latest features and fixes.
295 Change to the directory in which you want to put the source code, then
296 run:
297 \begin{verbatim}
298 # bk clone bk://xen.bkbits.net/xen-2.0.bk
299 \end{verbatim}
301 Under your current directory, a new directory named `xen-2.0.bk'
302 has been created, which contains all the source code for the Xen
303 hypervisor and the Xen tools. The directory also contains `sparse'
304 Linux source trees, containing only the files that differ between
305 XenLinux and standard Linux.
307 Once you have cloned the repository, you can update to the newest
308 changes to the repository by running:
309 \begin{verbatim}
310 # cd xen-2.0.bk # to change into the local repository
311 # bk pull # to update the repository
312 \end{verbatim}
314 \subsection{Without Bitkeeper}
316 The Xen source tree is also available in gzipped tarball form from the
317 Xen downloads page:\\
318 {\tt http://www.cl.cam.ac.uk/Research/SRG/netos/xen/downloads.html}.
319 Prebuilt tarballs are also available but are very large.
321 \section{The distribution}
323 The Xen source code repository is structured as follows:
325 \begin{description}
326 \item[\path{tools/}] Xen node controller daemon (Xend), command line tools,
327 control libraries
328 \item[\path{xen/}] The Xen hypervisor itself.
329 \item[\path{linux-2.4.27-xen/}] Linux 2.4 support for Xen
330 \item[\path{linux-2.6.7-xen/}] Linux 2.6 support for Xen
331 \item[\path{doc/}] various documentation files for users and developers
332 \item[\path{extras/}] currently this contains the Mini OS, aimed at developers
333 \end{description}
335 \section{Build and install}
337 The Xen makefile includes a target ``world'' that will do the
338 following:
340 \begin{itemize}
341 \item Build Xen
342 \item Build the control tools, including Xend
343 \item Download the ebtables patch for Linux 2.4
344 \item Download (if necessary) and unpack the Linux 2.4 source code,
345 and patch it for use with Xen
346 \item Build a Linux kernel to use in domain 0 and a smaller
347 unprivileged kernel, which can optionally be used for
348 unprivileged virtual machines.
349 \end{itemize}
351 Inspect the Makefile if you want to see what goes on during a
352 build. Building Xen and the tools is straightforward, but XenLinux is
353 more complicated. The makefile needs a `pristine' linux kernel tree
354 which it will then add the Xen architecture files to. You can tell the
355 makefile the location of the appropriate linux compressed tar file by
356 setting the LINUX\_SRC environment variable, e.g. \\
357 \verb!# LINUX_SRC=/tmp/linux-2.4.27.tar.gz make world! \\ or by
358 placing the tar file somewhere in the search path of {\tt LINUX\_SRC\_PATH}
359 which defaults to ``{\tt .:..}". If the makefile can't find a suitable
360 kernel tar file it attempts to download it from kernel.org (this won't
361 work if you're behind a firewall).
363 After untaring the pristine kernel tree, the makefile uses the {\tt
364 mkbuildtree} script to add the Xen patches the kernel. It then builds
365 two different XenLinux images, one with a ``-xen0'' extension which
366 contains hardware device drivers and is intended to be used in the
367 first virtual machine (``domain 0''), and one with a ``-xenU''
368 extension that just contains virtual-device drivers.
370 The procedure is similar to build the Linux 2.6 port: \\
371 \verb!# LINUX_SRC=/path/to/linux2.6/source make linux26!
373 In both cases, if you have an SMP machine you may wish to give the
374 {\tt '-j4'} argument to make to get a parallel build.
376 The files produced by the build process are stored under the
377 \path{install/} directory. To install them in their default
378 locations, do: \\
379 \verb_# make install_\\
381 Alternatively, users with special installation requirements may wish
382 to install them manually by copying file to their appropriate
383 destinations.
385 Take a look at the files in \path{install/boot/}:
386 \begin{itemize}
387 \item \path{install/boot/xen.gz} The Xen 'kernel'
388 \item \path{install/boot/vmlinuz-2.4.27-xen0} Domain 0 XenLinux kernel
389 \item \path{install/boot/vmlinuz-2.4.27-xenU} Unprivileged XenLinux kernel
390 \end{itemize}
392 The difference between the two Linux kernels that are built is due to
393 the configuration file used for each. The "U" suffixed unprivileged
394 version doesn't contain any of the physical hardware device drivers
395 --- it is 30\% smaller and hence may be preferred for your
396 non-privileged domains. The ``0'' suffixed privileged version can be
397 used to boot the system, as well as in driver domains and unprivileged
398 domains.
400 The \path{install/boot} directory will also contain the config files
401 used for building the XenLinux kernels, and also versions of Xen and
402 XenLinux kernels that contain debug symbols (\path{xen-syms} and
403 \path{vmlinux-syms-2.4.27-xen0}) which are essential for interpreting crash
404 dumps. Retain these files as the developers may wish to see them if
405 you post on the mailing list.
407 \section{Configuration}
409 \subsection{GRUB Configuration}
411 An entry should be added to \path{grub.conf} (often found under
412 \path{/boot/} or \path{/boot/grub/}) to allow Xen / XenLinux to boot.
413 This file is sometimes called \path{menu.lst}, depending on your
414 distribution. The entry should look something like the following:
416 \begin{verbatim}
417 title Xen 2.0 / XenLinux 2.4.27
418 kernel /boot/xen.gz dom0_mem=131072 com1=115200,8n1
419 module /boot/vmlinuz-2.4.27-xen0 root=/dev/sda4 ro console=tty0 console=ttyS0
420 \end{verbatim}
422 The first line of the configuration (kernel...) tells GRUB where to
423 find Xen itself and what boot parameters should be passed to it. The
424 second line of the configuration describes the location of the
425 XenLinux kernel that Xen should start and the parameters that should
426 be passed to it.
428 As always when installing a new kernel, it is recommended that you do
429 not remove the original contents of \path{grub.conf} --- you may want
430 to boot up with your old Linux kernel in future, particularly if you
431 have problems.
433 \subsection{Serial Console}
435 In order to configure serial console output, it is necessary to add a
436 line into \path{/etc/inittab}. The XenLinux console driver is
437 designed to make this procedure the same as configuring a normal
438 serial console. Add the line:
440 {\tt c:2345:respawn:/sbin/mingetty ttyS0}
442 \section{Test the new install}
444 It should now be possible to restart the system and use Xen. Reboot
445 as usual but choose the new Xen option when the Grub screen appears.
447 What follows should look much like a conventional Linux boot. The
448 first portion of the output comes from Xen itself, supplying low level
449 information about itself and the machine it is running on. The
450 following portion of the output comes from XenLinux itself.
452 You may see some errors during the XenLinux boot. These are not
453 necessarily anything to worry about --- they may result from kernel
454 configuration differences between your XenLinux kernel and the one you
455 usually use.
457 When the boot completes, you should be able to log into your system as
458 usual. If you are unable to log in to your system running Xen, you
459 should still be able to reboot with your normal Linux kernel.
462 \chapter{Starting a domain}
464 The first step in creating a new domain is to prepare a root
465 filesystem for it to boot off. Typically, this might be stored in a
466 normal partition, a disk file and LVM volume, or on an NFS server.
468 A simple way to do this is simply to boot from your standard OS
469 install CD and install the distribution into another partition on your
470 hard drive.
472 {\em N.b } you can boot with Xen and XenLinux without installing any
473 special userspace tools but will need to have the prerequisites
474 described in Section~\ref{sec:prerequisites} and the Xen control tools
475 are installed before you proceed.
477 \section{From the web interface}
479 Boot the Xen machine and start Xensv (see Chapter~\ref{cha:xensv} for
480 more details) using the command: \\
481 \verb_# xensv start_ \\
482 This will also start Xend (see Chapter~\ref{cha:xend} for more information).
484 The domain management interface will then be available at {\tt
485 http://your\_machine:8080/}. This provides a user friendly wizard for
486 starting domains and functions for managing running domains.
488 \section{From the command line}
490 Full details of the {\tt xm} tool are found in Chapter~\ref{cha:xm}.
492 This example explains how to use the \path{xmdefaults} file. If you
493 require a more complex setup, you will want to write a custom
494 configuration file --- details of the configuration file formats are
495 included in Chapter~\ref{cha:config}.
497 The \path{xmdefconfig1} file is a simple template configuration file
498 for describing a single VM.
500 The \path{xmdefconfig2} file is a template description that is intended
501 to be reused for multiple virtual machines. Setting the value of the
502 {\tt vmid} variable on the {\tt xm} command line
503 fills in parts of this template.
505 \subsection{Editing \path{xmdefconfig}}
507 At minimum, you should edit the following variables in \path{xmdefconfig}:
509 \begin{description}
510 \item[kernel] Set this to the path of the kernel you compiled for use
511 with Xen. [e.g. {\tt kernel =
512 '/root/xen-2.0.bk/install/boot/vmlinuz-2.4.27-xenU'}]
513 \item[memory] Set this to the size of the domain's memory in
514 megabytes. [e.g. {\tt memory = 64 } ]
515 \item[disk] Set the first entry in this list to calculate the offset
516 of the domain's root partition, based on the domain ID. Set the
517 second to the location of \path{/usr} (if you are sharing it between
518 domains). [i.e. {\tt disk = ['phy:your\_hard\_drive\%d,sda1,w' \%
519 (base\_partition\_number + vmid), 'phy:your\_usr\_partition,sda6,r' ]}
520 \item[dhcp] Uncomment the dhcp variable, so that the domain will
521 receive its IP address from a DHCP server. [i.e. {\tt dhcp=''dhcp''}]
522 \end{description}
524 You may also want to edit the {\bf vif} variable in order to choose
525 the MAC address of the virtual ethernet interface yourself. For
526 example: \\ \verb_vif = [`mac=00:06:AA:F6:BB:B3']_\\ If you do not set
527 this variable, Xend will automatically generate a random MAC address
528 from an unused range.
530 \subsection{Starting the domain}
532 The {\tt xm} tool provides a variety of commands for managing domains.
533 Use the {\tt create} command to start new domains. To start the
534 virtual machine with virtual machine ID 1.
536 \begin{verbatim}
537 # xm create -c vmid=1
538 \end{verbatim}
540 The {\tt -c} switch causes {\tt xm} to turn into the domain's console
541 after creation. The {\tt vmid=1} sets the {\tt vmid} variable used in
542 the {\tt xmdefconfig} file. The tool uses the
543 \path{/etc/xen/xmdefconfig} file, since no custom configuration file
544 was specified on the command line.
546 \chapter{Domain management tasks}
548 The previous chapter described a simple example of how to configure
549 and start a domain. This chapter summarises the tools available to
550 manage running domains.
552 \section{Command line management}
554 Command line management tasks are also performed using the {\tt xm}
555 tool. For online help for the commands available, type:\\
556 \verb_# xm help_
558 \subsection{Basic management commands}
560 The most important {\tt xm} commands are: \\
561 \verb_# xm list_ : Lists all domains running. \\
562 \verb_# xm consoles_ : Gives information about the domain consoles. \\
563 \verb_# xm console_: open a console to a domain.
564 e.g. \verb_# xm console 1_ (open console to domain 1)
566 \subsection{\tt xm list}
568 The output of {\tt xm list} is in rows of the following format:\\
569 \verb_domid name memory cpu state cputime_
571 \begin{description}
572 \item[domid] The number of the domain ID this virtual machine is running in.
573 \item[name] The descriptive name of the virtual machine.
574 \item[memory] Memory size in megabytes.
575 \item[cpu] The CPU this domain is running on.
576 \item[state] Domain state consists of 5 fields:
577 \begin{description}
578 \item[r] running
579 \item[b] blocked
580 \item[p] paused
581 \item[s] shutdown
582 \item[c] crashed
583 \end{description}
584 \item[cputime] How much CPU time (in seconds) the domain has used so far.
585 \end{description}
587 The {\tt xm list} command also supports a long output format when the
588 {\tt -l} switch is used. This outputs the fulls details of the
589 running domains in Xend's SXP configuration format.
591 \chapter{Other kinds of storage}
593 It is possible to use any Linux block device to store virtual machine
594 disk images. This chapter covers some of the possibilities; note that
595 it is also possible to use network-based block devices and other
596 unconventional block devices.
598 \section{File-backed virtual block devices}
600 It is possible to use a file in Domain 0 as the primary storage for a
601 virtual machine. As well as being convenient, this also has the
602 advantage that the virtual block device will be {\em sparse} --- space
603 will only really be allocated as parts of the file are used. So if a
604 virtual machine uses only half its disk space then the file really
605 takes up a half of the size allocated.
607 For example, to create a 2GB sparse file-backed virtual block device
608 (actually only consumes 1KB of disk):
610 \verb_# dd if=/dev/zero of=vm1disk bs=1k seek=2048k count=1_
612 Choose a free loop back device, and attach file: \\
613 \verb_# losetup /dev/loop0 vm1disk_ \\
614 Make a file system on the loop back device: \\
615 \verb_# mkfs -t ext3 /dev/loop0_
617 Populate the file system e.g. by copying from the current root:
618 \begin{verbatim}
619 # mount /dev/loop0 /mnt
620 # cp -ax / /mnt
621 \end{verbatim}
622 Tailor the file system by editing \path{/etc/fstab},
623 \path{/etc/hostname}, etc (don't forget to edit the files in the
624 mounted file system, instead of your domain 0 filesystem, e.g. you
625 would edit \path{/mnt/etc/fstab} instead of \path{/etc/fstab} ). For
626 this example put \path{/dev/sda1} to root in fstab.
628 Now unmount (this is important!):\\
629 \verb_# umount /dev/loop0_
631 In the configuration file set:\\
632 \verb_disk = [`phy:loop0,sda1,w']_
634 As the virtual machine writes to its `disk', the sparse file will be
635 filled in and consume more space up to the original 2GB.
637 {\em NB.} You will need to use {\tt losetup} to bind the file to
638 \path{/dev/loop0} (or whatever loopback device you chose) each time
639 you reboot domain 0. In the near future, Xend will track which loop
640 devices are currently free and do binding itself, making this manual
641 effort unnecessary.
643 \section{LVM-backed virtual block devices}
645 XXX Put some simple examples here - would be nice if an LVM user could
646 contribute some, although obviously users would have to read the LVM
647 docs to do advanced stuff.
649 \part{Quick Reference}
651 \chapter{Domain Configuration Files}
652 \label{cha:config}
654 XXX Could use a little explanation about possible values
656 Xen configuration files contain the following standard variables:
658 \begin{description}
659 \item[kernel] Path to the kernel image (on the server).
660 \item[ramdisk] Path to a ramdisk image (optional).
661 \item[builder] The name of the domain build function (e.g. {\tt'linux'} or {\tt'netbsd'}.
662 \item[memory] Memory size in megabytes.
663 \item[cpu] CPU to assign this domain to.
664 \item[nics] Number of virtual network interfaces.
665 \item[vif] List of MAC addresses (random addresses are assigned if not given).
666 \item[disk] Regions of disk to export to the domain.
667 \item[dhcp] Set to {\tt 'dhcp'} if you want to DHCP allocate the IP address.
668 \item[netmask] IP netmask.
669 \item[gateway] IP address for the gateway (if any).
670 \item[hostname] Set the hostname for the virtual machine.
671 \item[root] Set the root device.
672 \item[nfs\_server] IP address for the NFS server.
673 \item[nfs\_root] Path of the root filesystem on the NFS server.
674 \item[extra] Extra string to append to the kernel command line.
675 \item[restart] Three possible options:
676 \begin{description}
677 \item[always] Always restart the domain, no matter what
678 its exit code is.
679 \item[never] Never restart the domain.
680 \item[onreboot] (restart the domain if it requests reboot).
681 \end{description}
682 \end{description}
684 It is also possible to include Python scripting commands in
685 configuration files. This is done in the \path{xmdefconfig} file in
686 order to handle the {\tt vmid} variable.
689 \chapter{Xend (Node control daemon)}
690 \label{cha:xensv}
692 The Xen Daemon (Xend) performs system management functions related to
693 virtual machines. It forms a central point of control for a machine
694 and can be controlled using an HTTP-based protocol. Xend must be
695 running in order to start and manage virtual machines.
697 Xend must be run as root because it needs access to privileged system
698 management functions. A small set of commands may be issued on the
699 Xend command line:
701 \begin{tabular}{ll}
702 \verb_# xend start_ & start Xend, if not already running \\
703 \verb_# xend stop_ & stop Xend if already running \\
704 \verb_# xend restart_ & restart Xend if running, otherwise start it \\
705 \end{tabular}
707 An SysV init script called {\tt xend} is provided to start Xend at
708 boot time. The {\tt make install} will install this script in
709 {\path{/etc/init.d} automatically. To enable it, you can make
710 symbolic links in the appropriate runlevel directories or use the {\tt
711 chkconfig} tool, where available.
713 Once Xend is running, more sophisticated administration can be done
714 using the Xensv web interface (see Chapter~\ref{cha:xensv}).
716 \chapter{Xensv (Web interface server)}
717 \label{cha:xensv}
719 Xensv is the server for the web control interface. It can be started
720 using:\\
721 \verb_# xensv start_ \\
722 and stopped using:
723 \verb_# xensv stop_ \\
724 It will automatically start Xend if it is not already running.
726 By default, Xensv will serve out the web interface on port 8080. This
727 can be changed by editing {\tt
728 /usr/lib/python2.2/site-packages/xen/sv/params.py}.
730 Once Xensv is running, the web interface can be used to manage running
731 domains and provides a user friendly domain creation wizard.
733 \chapter{The xm tool}
734 \label{cha:xm}
736 XXX Add description of arguments and switches for all the options
738 The xm tool is the primary tool for managing Xen from the console.
739 The general format of an xm command line is:
741 \begin{verbatim}
742 # xm command [switches] [arguments] [variables]
743 \end{verbatim}
745 The available {\em switches } and {\em arguments}are dependent on the
746 {\em command} chosen. The {\em variables} may be set using
747 declarations of the form {\tt variable=value} and may be used to set /
748 override any of the values in the configuration file being used,
749 including the standard variables described above and any custom
750 variables (for instance, the \path{xmdefconfig} file uses a {\tt vmid}
751 variable).
753 The available commands are as follows:
755 \begin{description}
756 \item[create] Create a new domain.
757 \item[destroy] Kill a domain immediately.
758 \item[list] List running domains.
759 \item[shutdown] Ask a domain to shutdown.
760 \item[dmesg] Fetch the Xen (not Linux!) boot output.
761 \item[consoles] Lists the available consoles.
762 \item[console] Connect to the console for a domain.
763 \item[help] Get help on xm commands.
764 \item[save] Suspend a domain to disk.
765 \item[restore] Restore a domain from disk.
766 \item[pause] Pause a domain's execution.
767 \item[unpause] Unpause a domain.
768 \item[pincpu] Pin a domain to a CPU.
769 \item[bvt] Set BVT scheduler parameters for a domain.
770 \item[bvt\_ctxallow] Set the BVT context switching allowance for the system.
771 \item[fbvt] Set the FBVT scheduler parameters for a domain.
772 \item[fbvt\_ctxallow] Set the FBVT context switching allowance for the system.
773 \item[atropos] Set the atropos parameters for a domain.
774 \item[rrobin] Set the round robin time slice for the system.
775 \item[info] Get information about the Xen host.
776 \item[call] Call a Xend HTTP API function directly.
777 \end{description}
779 \chapter{Glossary}
781 \begin{description}
782 \item[Atropos] One of the CPU schedulers provided by Xen.
783 Atropos provides domains with absolute shares
784 of the CPU, with timeliness guarantees and a
785 mechanism for sharing out ``slack time''.
787 \item[BVT] The BVT scheduler is used to give proportional
788 fair shares of the CPU to domains.
790 \item[Exokernel] A minimal piece of privileged code, similar to
791 a {\bf microkernel} but providing a more
792 `hardware-like' interface to the tasks it
793 manages. This is similar to a paravirtualising
794 VMM like {\bf Xen} but was designed as a new
795 operating system structure, rather than
796 specifically to run multiple conventional OSs.
798 \item[FBVT] A derivative of the { \bf BVT } scheduler that
799 aims to give better fairness performance to IO
800 intensive domains in competition with CPU
801 intensive domains.
803 \item[Domain] A domain is the execution context that
804 contains a running { \bf virtual machine }.
805 The relationship between virtual machines
806 and domains on Xen is similar to that between
807 programs and processes in an operating
808 system: a virtual machine is a persistent
809 entity that resides on disk (somewhat like
810 a program). When it is loaded for execution,
811 it runs in a domain. Each domain has a
812 { \bf domain ID }.
814 \item[Domain 0] The first domain to be started on a Xen
815 machine. Domain 0 is responsible for managing
816 the system.
818 \item[Domain ID] A unique identifier for a { \bf domain },
819 analogous to a process ID in an operating
820 system. Apart from domain
822 \item[Full virtualisation] An approach to virtualisation which
823 requires no modifications to the hosted
824 operating system, providing the illusion of
825 a complete system of real hardware devices.
827 \item[Hypervisor] An alternative term for { \bf VMM }, used
828 because it means ``beyond supervisor'',
829 since it is responsible for managing multiple
830 ``supervisor'' kernels.
832 \item[Microkernel] A small base of code running at the highest
833 hardware privilege level. A microkernel is
834 responsible for sharing CPU and memory (and
835 sometimes other devices) between less
836 privileged tasks running on the system.
837 This is similar to a VMM, particularly a
838 {\bf paravirtualising} VMM but typically
839 addressing a different problem space and
840 providing different kind of interface.
842 \item[NetBSD/Xen] A port of NetBSD to the Xen architecture.
844 \item[Paravirtualisation] An approach to virtualisation which requires
845 modifications to the operating system in
846 order to run in a virtual machine. Xen
847 uses paravirtualisation but preserves
848 binary compatibility for user space
849 applications.
851 \item[Virtual Machine] The environment in which a hosted operating
852 system runs, providing the abstraction of a
853 dedicated machine. A virtual machine may
854 be identical to the underlying hardware (as
855 in { \bf full virtualisation }, or it may
856 differ, as in { \bf paravirtualisation }.
858 \item[VMM] Virtual Machine Monitor - the software that
859 allows multiple virtual machines to be
860 multiplexed on a single physical machine.
862 \item[Xen] Xen is a paravirtualising virtual machine
863 monitor, developed primarily by the
864 Systems Research Group at the University
865 of Cambridge Computer Laboratory.
867 \item[XenLinux] Official name for the port of the Linux kernel
868 that runs on Xen.
870 \end{description}
872 \part{Advanced Topics}
874 XXX More to add here, including config file format
876 \chapter{Advanced Network Configuration}
878 For simple systems with a single ethernet interface with a simple
879 configuration, the default installation should work ``out of the
880 box''. More complicated network setups, for instance with multiple
881 ethernet interfaces and / or existing bridging setups will require
882 some special configuration.
884 The purpose of this chapter is to describe the mechanisms provided by
885 xend to allow a flexible configuration for Xen's virtual networking.
887 \section{Xen networking scripts}
889 Xen's virtual networking is configured by 3 shell scripts. These are
890 called automatically by Xend when certain events occur, with arguments
891 to the scripts providing further contextual information. These
892 scripts are found by default in \path{/etc/xen}. The names and
893 locations of the scripts can be configured in \path{xend-config.sxp}.
895 \subsection{\path{network}}
897 This script is called once when Xend is started and once when Xend is
898 stopped. Its job is to do any advance preparation required for the
899 Xen virtual network when Xend starts and to do any corresponding
900 cleanup when Xend exits.
902 In the default configuration, this script creates the bridge
903 ``xen-br0'' and moves eth0 onto that bridge, modifying the routing
904 accordingly.
906 In configurations where the bridge already exists, this script could
907 be replaced with a link to \path{/bin/true} (for instance).
909 When Xend exits, this script is called with the {\tt stop} argument,
910 which causes it to delete the Xen bridge and remove {\tt eth0} from
911 it, restoring the normal IP and routing configuration.
913 \subsection{\path{vif-bridge}}
915 This script is called for every domain virtual interface. This should
916 do things like configuring firewalling rules for that interface and
917 adding it to the appropriate bridge.
919 By default, this adds and removes VIFs on the default Xen bridge.
920 This script can be customized to properly deal with more complicated
921 bridging setups.
923 \chapter{Advanced Scheduling Configuration}
925 \section{Scheduler selection}
927 Xen offers a boot time choice between multiple schedulers. To select
928 a scheduler, pass the boot parameter { \tt sched=sched\_name } to Xen,
929 substituting the appropriate scheduler name. Details of the schedulers
930 and their parameters are included below; future versions of the tools
931 will provide a higher-level interface to these tools.
933 \section{Borrowed Virtual Time}
935 BVT provides proportional fair shares of the CPU time. It has been
936 observed to penalise domains that block frequently (e.g. IO intensive
937 domains), so the FBVT derivative has been included as an alternative.
939 \subsection{Global Parameters}
941 \begin{description}
942 \item[ctx\_allow]
943 the context switch allowance is similar to the "quantum"
944 in traditional schedulers. It is the minimum time that
945 a scheduled domain will be allowed to run before be
946 pre-empted. This prevents thrashing of the CPU.
947 \end{description}
949 \subsection{Per-domain parameters}
951 \begin{description}
952 \item[mcuadv]
953 the MCU (Minimum Charging Unit) advance determines the
954 proportional share of the CPU that a domain receives. It
955 is set inversely proportionally to a domain's sharing weight.
956 \item[warp]
957 the amount of "virtual time" the domain is allowed to warp
958 backwards
959 \item[warpl]
960 the warp limit is the maximum time a domain can run warped for
961 \item[warpu]
962 the unwarp requirement is the minimum time a domain must
963 run unwarped for before it can warp again
964 \end{description}
966 \section{Fair Borrowed Virtual Time}
968 This is a derivative for BVT that aims to provide better fairness for
969 IO intensive domains as well as for CPU intensive domains.
971 \subsection{Global Parameters}
973 Same as for BVT.
975 \subsection{Per-domain parameters}
977 Same as for BVT.
979 \section{Atropos}
981 Atropos is a Soft Real Time scheduler. It provides guarantees about
982 absolute shares of the CPU (with a method for optionally sharing out
983 slack CPU time on a best-effort basis) and can provide timeliness
984 guarantees for latency-sensitive domains.
986 \subsection{Per-domain parameters}
988 \begin{description}
989 \item[slice]
990 The length of time per period that a domain is guaranteed.
991 \item[period]
992 The period over which a domain is guaranteed to receive
993 its slice of CPU time.
994 \item[latency]
995 The latency hint is used to control how soon after
996 waking up a domain should be scheduled.
997 \item[xtratime]
998 This is a true (1) / false (0) flag that specifies whether
999 a domain should be allowed a share of the system slack time.
1000 \end{description}
1002 \section{Round Robin}
1004 The Round Robin scheduler is included as a simple demonstration of
1005 Xen's internal scheduler API. It is not intended for production use
1006 --- the other schedulers included are all more general and should give
1007 higher throughput.
1009 \subsection{Global parameters}
1011 \begin{description}
1012 \item[rr\_slice]
1013 The maximum time each domain runs before the next
1014 scheduling decision is made.
1015 \end{description}
1017 \chapter{Privileged domains}
1019 There are two possible types of privileges: IO privileges and
1020 administration privileges.
1022 \section{Driver domains (IO Privileges)}
1024 IO privileges can be assigned to allow a domain to drive PCI devices
1025 itself. This is used for to support driver domains.
1027 Setting backend privileges is currently only supported in SXP format
1028 config files (??? is this true - there's nothing in xmdefconfig,
1029 anyhow). To allow a domain to function as a backend for others,
1030 somewhere within the {\tt vm} element of its configuration file must
1031 be a {\tt backend} element of the form {\tt (backend ({\em type}))}
1032 where {\tt \em type} may be either {\tt netif} or {\tt blkif},
1033 according to the type of virtual device this domain will service.
1034 After this domain has been built, Xend will connect all new and
1035 existing {\em virtual} devices (of the appropriate type) to that
1036 backend.
1038 Note that:
1039 \begin{itemize}
1040 \item a block backend cannot import virtual block devices from other
1041 domains
1042 \item a network backend cannot import virtual network devices from
1043 other domains
1044 \end{itemize}
1046 Thus (particularly in the case of block backends, which cannot import
1047 a virtual block device as their root filesystem), you may need to boot
1048 a backend domain from a ramdisk or a network device.
1050 The privilege to drive PCI devices may also be specified on a
1051 per-device basis. Xen will assign the minimal set of hardware
1052 privileges to a domain that are required to control its devices. This
1053 can be configured in either format of configuration file:
1055 \begin{itemize}
1056 \item SXP Format:
1057 Include {\tt device} elements
1058 {\tt (device (pci (bus {\em x}) (dev {\em y}) (func {\em z}))) } \\
1059 inside the top-level {\tt vm} element. Each one specifies the address
1060 of a device this domain is allowed to drive ---
1061 the numbers {\em x},{\em y} and {\em z} may be in either decimal or
1062 hexadecimal format.
1063 \item Flat Format: Include a list of PCI device addresses of the
1064 format: \\ {\tt pci = ['x,y,z', ...] } \\ where each element in the
1065 list is a string specifying the components of the PCI device
1066 address, separated by commas. The components ({\tt \em x}, {\tt \em
1067 y} and {\tt \em z}) of the list may be formatted as either decimal
1068 or hexadecimal.
1069 \end{itemize}
1071 \section{Administration Domains}
1073 Administration privileges allow a domain to use the ``dom0
1074 operations'' (so called because they are usually available only to
1075 domain 0). A privileged domain can build other domains, set scheduling
1076 parameters, etc.
1078 % Support for other administrative domains is not yet available...
1080 \chapter{Xen build options}
1082 For most users, the default build of Xen will be adequate. For some
1083 advanced uses, Xen provides a number of build-time options:
1085 At build time, these options should be set as environment variables or
1086 passed on make's command-line. For example:
1088 \begin{verbatim}
1089 export option=y; make
1090 option=y make
1091 make option1=y option2=y
1092 \end{verbatim}
1094 \section{List of options}
1096 {\bf debug=y }\\
1097 Enable debug assertions and console output.
1098 (Primarily useful for tracing bugs in Xen). \\
1099 {\bf debugger=y }\\
1100 Enable the in-Xen pervasive debugger (PDB).
1101 This can be used to debug Xen, guest OSes, and
1102 applications. For more information see the
1103 XenDebugger-HOWTO. \\
1104 {\bf perfc=y }\\
1105 Enable performance-counters for significant events
1106 within Xen. The counts can be reset or displayed
1107 on Xen's console via console control keys. \\
1108 {\bf trace=y }\\
1109 Enable per-cpu trace buffers which log a range of
1110 events within Xen for collection by control
1111 software. For more information see the chapter on debugging,
1112 in the Xen Interface Manual.
1114 \chapter{Xen boot options}
1116 These options are used to configure Xen's behaviour at runtime. They
1117 should be appended to Xen's command line, either manually or by
1118 editing \path{grub.conf}.
1120 \section{List of options}
1122 {\bf ignorebiostables }\\
1123 Disable parsing of BIOS-supplied tables. This may help with some
1124 chipsets that aren't fully supported by Xen. If you specify this
1125 option then ACPI tables are also ignored, and SMP support is
1126 disabled. \\
1128 {\bf noreboot } \\
1129 Don't reboot the machine automatically on errors. This is
1130 useful to catch debug output if you aren't catching console messages
1131 via the serial line. \\
1133 {\bf nosmp } \\
1134 Disable SMP support.
1135 This option is implied by 'ignorebiostables'. \\
1137 {\bf noacpi } \\
1138 Disable ACPI tables, which confuse Xen on some chipsets.
1139 This option is implied by 'ignorebiostables'. \\
1141 {\bf watchdog } \\
1142 Enable NMI watchdog which can report certain failures. \\
1144 {\bf noht } \\
1145 Disable Hyperthreading. \\
1147 {\bf badpage=$<$page number$>$[,$<$page number$>$] } \\
1148 Specify a list of pages not to be allocated for use
1149 because they contain bad bytes. For example, if your
1150 memory tester says that byte 0x12345678 is bad, you would
1151 place 'badpage=0x12345' on Xen's command line (i.e., the
1152 last three digits of the byte address are not
1153 included!). \\
1155 {\bf com1=$<$baud$>$,DPS[,$<$io\_base$>$,$<$irq$>$] \\
1156 com2=$<$baud$>$,DPS[,$<$io\_base$>$,$<$irq$>$] } \\
1157 Xen supports up to two 16550-compatible serial ports.
1158 For example: 'com1=9600,8n1,0x408,5' maps COM1 to a
1159 9600-baud port, 8 data bits, no parity, 1 stop bit,
1160 I/O port base 0x408, IRQ 5.
1161 If the I/O base and IRQ are standard (com1:0x3f8,4;
1162 com2:0x2f8,3) then they need not be specified. \\
1164 {\bf console=$<$specifier list$>$ } \\
1165 Specify the destination for Xen console I/O.
1166 This is a comma-separated list of, for example:
1167 \begin{description}
1168 \item[vga] use VGA console and allow keyboard input
1169 \item[com1] use serial port com1
1170 \item[com2H] use serial port com2. Transmitted chars will
1171 have the MSB set. Received chars must have
1172 MSB set.
1173 \item[com2L] use serial port com2. Transmitted chars will
1174 have the MSB cleared. Received chars must
1175 have MSB cleared.
1176 \end{description}
1177 The latter two examples allow a single port to be
1178 shared by two subsystems (e.g. console and
1179 debugger). Sharing is controlled by MSB of each
1180 transmitted/received character.
1181 [NB. Default for this option is 'com1,tty'] \\
1183 {\bf conswitch=$<$switch-char$><$auto-switch-char$>$ } \\
1184 Specify how to switch serial-console input between
1185 Xen and DOM0. The required sequence is CTRL-<switch-char>
1186 pressed three times. Specifying '`' disables switching.
1187 The <auto-switch-char> specifies whether Xen should
1188 auto-switch input to DOM0 when it boots -- if it is 'x'
1189 then auto-switching is disabled. Any other value, or
1190 omitting the character, enables auto-switching.
1191 [NB. Default for this option is 'a'] \\
1193 {\bf nmi=xxx } \\
1194 Specify what to do with an NMI parity or I/O error. \\
1195 'nmi=fatal': Xen prints a diagnostic and then hangs. \\
1196 'nmi=dom0': Inform DOM0 of the NMI. \\
1197 'nmi=ignore': Ignore the NMI. \\
1199 {\bf dom0\_mem=xxx } \\
1200 Set the maximum amount of memory for domain0. \\
1202 {\bf tbuf\_size=xxx } \\
1203 Set the size of the per-cpu trace buffers, in pages
1204 (default 1). Note that the trace buffers are only
1205 enabled in debug builds. Most users can ignore
1206 this feature completely. \\
1208 {\bf sched=xxx } \\
1209 Select the CPU scheduler Xen should use. The current
1210 possibilities are 'bvt', 'atropos' and 'rrobin'. The
1211 default is 'bvt'. For more information see
1212 Sched-HOWTO.txt. \\
1214 {\bf pci\_dom0\_hide=(xx.xx.x)(yy.yy.y)... } \\
1215 Hide selected PCI devices from domain 0 (for instance, to stop it
1216 taking ownership of them so that they can be driven by another
1217 domain). Device IDs should be given in hex format. Bridge devices do
1218 not need to be hidden --- they are hidden implicitly, since guest OSes
1219 do not need to configure them.
1221 \chapter{Further Support}
1223 If you have questions that are not answered by this manual, the
1224 sources of information listed below may be of interest to you. Note
1225 that bug reports, suggestions and contributions related to the
1226 software (or the documentation) should be sent to the Xen developers'
1227 mailing list (address below).
1229 \section{Other documentation}
1231 For developers interested in porting operating systems to Xen, the
1232 {\em Xen Interface Manual} is distributed in the \path{docs/}
1233 directory of the Xen source distribution. Various HOWTOs are
1234 available in \path{docs/HOWTOS} but this content is being integrated
1235 into this manual.
1237 \section{Online references}
1239 The official Xen web site is found at: \\
1240 {\tt
1241 http://www.cl.cam.ac.uk/Research/SRG/netos/xen/] }.
1243 Links to other
1244 documentation sources are listed at: \\ {\tt
1245 http://www.cl.cam.ac.uk/Research/SRG/netos/xen/documentation.html}.
1247 \section{Mailing lists}
1249 There are currently two official Xen mailing lists:
1251 \begin{description}
1252 \item[xen-devel@lists.sourceforge.net] Used for development
1253 discussions and requests for help. Subscribe at: \\
1254 {\tt http://lists.sourceforge.net/mailman/listinfo/xen-devel}
1255 \item[xen-announce@lists.sourceforge.net] Used for announcements only.
1256 Subscribe at: \\
1257 {\tt http://lists.sourceforge.net/mailman/listinfo/xen-announce}
1258 \end{description}
1260 Although there is no specific user support list, the developers try to
1261 assist users who post on xen-devel. As the bulk of traffic on this
1262 list increases, a dedicated user support list may be introduced.
1264 \end{document}