view README.CD @ 747:885d2d2dbb61

bitkeeper revision 1.447 (3f6b4a5bx66eWPkM9ZVWGugRdLunrg)

author iap10@labyrinth.cl.cam.ac.uk
date Fri Sep 19 18:26:35 2003 +0000 (2003-09-19)
parents 57012a0c028c
children e0274a094846
line source
1 #############################
2 __ __ _ ___
3 \ \/ /___ _ __ / | / _ \
4 \ // _ \ '_ \ | || | | |
5 / \ __/ | | | | || |_| |
6 /_/\_\___|_| |_| |_(_)___/
8 #############################
10 XenDemoCD 1.0 rc1
11 University of Cambridge Computer Laboratory
12 18 Sep 2003
14 http://www.cl.cam.ac.uk/netos/xen
16 Welcome to the Xen Demo CD!
18 Executive Summary
19 =================
21 This CD is a standalone demo of the Xen Virtual Machine Monitor (VMM)
22 and Linux-2.4 OS port (XenoLinux). It runs entirely off the CD,
23 without requiring hard disk installation. This is achieved using a RAM
24 disk to store mutable file system data while using the CD for
25 everything else. The CD can also be used for installing Xen/XenoLinux
26 to disk, and includes a source code snapshot along with all of the
27 tools required to build it.
29 Booting the CD
30 ==============
32 The Xen VMM is currently fairly h/w specific, but porting new device
33 drivers is relatively straightforward thanks to Xen's Linux driver
34 compatibility layer. The current snapshot supports the following
35 hardware:
37 CPU: Pentium Pro/II/III/IV/Xeon, Athlon (i.e. P6 or newer) SMP supported
38 IDE: Intel PIIX chipset, others will be PIO only (slow)
39 SCSI: Adaptec / Dell PERC Raid (aacraid), megaraid, Adaptec aic7xxx
40 Net: Recommended: Intel e1000, Broadcom BCM57xx (tg3), 3c905 (3c59x)
41 Tested but require extra copies : pcnet32, Intel e100
42 Untested and also requires extra copies : tulip
44 Because of the demo CD's use of RAM disks, make sure you have plenty
45 of RAM (256MB+).
47 To try out the Demo, boot from CD (you may need to change your BIOS
48 configuration to do this), hit a key on either the keyboard or serial
49 line to pull up the Grub boot menu, then select one of the three boot
50 options:
52 Xen / linux-2.4.22
53 Xen / linux-2.4.22 using cmdline IP configuration
54 linux-2.4.22
56 The last option is a plain linux kernel that runs on the bare machine,
57 and is included simply to help diagnose driver compatibility
58 problems. If you are going for a command line IP config, hit "e" at
59 the grub menu, then edit the "ip=" parameters to reflect your setup
60 e.g. "ip=<ipaddr>::<gateway>:<netmask>::eth0:off". It shouldn't be
61 necessary to set either the nfs server or hostname
62 parameters. Alternatively, once XenoLinux has booted you can login and
63 setup networking with 'dhclient' or 'ifconfig' and 'route' in the
64 normal way.
66 To make things easier for yourself, it's worth trying to arrange for an
67 IP address which is the first in a sequential range of free IP
68 addresses. It's useful to give each VM instance its own public IP
69 address (though it is possible to do NAT or use private addresses),
70 and the configuration files on the CD allocate IP addresses
71 sequentially for subsequent domains unless told otherwise.
73 After selecting the kernel to boot, stand back and watch Xen boot,
74 closely followed by "domain 0" running the XenoLinux kernel. The boot
75 messages are also sent to the serial line (the baud rate can be set on
76 the Xen cmdline, but defaults to 115200), which can be very useful for
77 debugging should anything important scroll off the screen. Xen's
78 startup messages will look quite familiar as much of the hardware
79 initialisation (SMP boot, apic setup) and device drivers are derived
80 from Linux.
82 If everything is well, you should see the linux rc scripts start a
83 bunch of standard services including sshd. Login on the console or
84 via ssh::
85 username: user root
86 password: xendemo xendemo
88 Once logged in, it should look just like any regular linux box. All
89 the usual tools and commands should work as per usual. It's probably
90 best to start by configuring networking, either with 'dhclient' or
91 manually via ifconfig and route, remembering to edit /etc/resolv.conf
92 if you want DNS.
94 You can start an X server with 'startx'. It defaults to a conservative
95 1024x768, but you can edit the script for higher resoloutions. The CD
96 contains a load of standard software. You should be able to start
97 Apache, PostgreSQL, Mozzila etc in the normal way, but because
98 everything is running off CD the performance will be very sluggish and
99 you may run out of memory for the 'tmpfs' file system. You may wish
100 to go ahead and install Xen/XenoLinux on your hard drive, either
101 dropping Xen and the XenoLinux kernel down onto a pre-existing Linux
102 distribution, or using the file systems from the CD (which are based
103 on RH9). See the installation instructions later in this document.
105 If you want to browse the Xen / XenoLinux source, it's all located
106 under /usr/local/src, complete with BitKeeper repository. We've also
107 included source code and configuration information for the various
108 benchmarks we used in the SOSP paper.
111 Starting other domains
112 ======================
114 There's a web interface for starting and managing other domains (VMs),
115 but since we generally use the command line tools they're probably
116 rather better debugged at present. The key command is 'xenctl' which
117 lives in /usr/local/bin and uses /etc/xenctl.xml for its default
118 configuration. Run 'xenctl' without any arguments to get a help
119 message. Note that xenctl is a java front end to various underlying
120 internal tools written in C (xi_*). Running off CD, it seems to take
121 an age to start...
123 Abyway, the first thing to do is to set up a window in which you will
124 receive console output from other domains. Console output will arrive
125 as UDP packets destined for, so its necessary to setup an
126 alias on eth0. The easiest way to do this is to run:
128 xen_nat_enable
130 This also inserts a few NAT rules into "domain0", in case you'll be
131 starting other domains without their own IP addresses. Alternatively,
132 just do "ifconfig eth0:0 up". NB: The intention is that in
133 future Xen will do NAT itsel (actually RSIP), but this is part of a
134 larger work package that isn't stable enough to release.
136 Next, run a the xen UDP console displayer:
138 xen_read_console &
141 As mentioned above, xenctl uses /etc/xenctl.xml as its default
142 configuration. The directory contains two different configs depending
143 on whether you want to use NAT, or multiple sequential external IPs
144 (it's possible to override any of the parameters on the command line
145 if you want to set specific IPs, etc).
147 The default configuration file supports NAT. To change to use multiple IPs:
149 cp /etc/xenctl.xml-publicip /etc/xenctl.xml
151 A sequence of commands must be given to xenctl to start a new
152 domain. First a new domain must be created, which requires specifying
153 the initial memory allocation, the kernel image to use, and the kernel
154 command line. As well as the root file system details, you'll need to
155 set the IP address on the command line: since Xen currently doesn't
156 support a virtual console for domains >1, you won't be able to log to
157 your new domain unless you've got networking configured and an sshd
158 running! (using dhcp for new domains should work too).
160 After creating the domain, xenctl must be used to grant the domain
161 access to other resources such as physical or virtual disk partions.
162 Then, the domain must be started.
164 These commands can be entered manually, but for convenience, xenctl
165 will also read them from a script and infer which domain number you're
166 referring to (-nX). To use the sample script:
168 xenctl script -f/etc/xen-mynewdom [NB: no space after the -f]
170 You should see the domain booting on your xen_read_console window.
172 The xml defaults start another domain running off the CD, using a
173 separate RAM-based file system for mutable data in root (just like
174 domain 0).
176 The new domain is started with a '4' on the kernel command line to
177 tell 'init' to go to runlevel 4 rather than the default of 3. This is
178 done simply to suppress a bunch of harmless error messages that would
179 otherwise occur when the new (unprivileged) domain tried to access
180 physical hardware resources to try setting the hwclock, system font,
181 run gpm etc.
183 After it's booted, you should be able to ssh into your new domain. If
184 you went for a NATed address, from domain 0 you should be able to ssh
185 into '169.254.1.X' where X is the domain number. If you ran the
186 xen_enable_nat script, a bunch of port redirects have been installed
187 to enable you to ssh in to other domains remotely. To access the new
188 virtual machine remotely, use:
190 ssh -p2201 root@IP.address.Of.Domain0 # use 2202 for domain 2 etc.
192 If you configured the new domain with its own IP address, you should
193 be able to ssh into it directly.
196 "xenctl domain list" provides status information about running domains,
197 though is currently only allowed to be run by domain 0. It accesses
198 /proc/xeno/domains to read this information from Xen. You can also use
199 xenctl to 'stop' (pause) a domain, or 'kill' a domain. You can either
200 kill it nicely by sending a shutdown event and waiting for it to
201 terminate, or blow the sucker away with extreme prejudice.
203 If you want to configure a new domain differently, type 'xenctl' to
204 get a list of arguments, e.g. at the 'xenctl domain new' command line
205 use the "-4" option to set a diffrent IPv4 address.
207 xenctl can be used to set the new kernel's command line, and hence
208 determine what it uses as a root file system, etc. Although the default
209 is to boot in the same manner that domain0 did (using the RAM-based
210 file system for root and the CD for /usr) it's possible to configure any
211 of the following possibilities, for example:
213 * initrd=/boot/initrd init=/linuxrc
214 boot using an initial ram disk, executing /linuxrc (as per this CD)
216 * root=/dev/hda3 ro
217 boot using a standard hard disk partition as root
219 * root=/dev/xvda1 ro
220 boot using a pre-configured 'virtual block device' that will be
221 attached to a virtual disk that previously has had a file system
222 installed on it.
224 * root=/dev/nfs nfsroot=/path/on/server ip=<blah_including server_IP>
225 Boot using an NFS mounted root file system. This could be from a
226 remote NFS server, or from an NFS server running in another
227 domain. The latter is rather a useful option.
230 A typical setup might be to allocate a standard disk partition for
231 each domain and populate it with files. To save space, having a shared
232 read-only usr partition might make sense.
234 Alternatively, you can use 'virtual disks', which are stored as files
235 within a custom file system. "xenctl partitions add" can be used to
236 'format' a partition with the file system, and then virtual disks can
237 be created with "xenctl vd create". Virtual disks can then be attached
238 to a running domain as a 'virtual block device' using "xenctl vdb
239 create". The virtual disk can optionally be partitioned (e.g. "fdisk
240 /dev/xvda") or have a file system created on it directly (e.g. "mkfs
241 -t ext3 /dev/xvda"). The virtual disk can then be accessed by a
242 virtual block device associated with another domain, and even used as
243 a boot device.
245 Both virtual disks and real partitions should only be shared between
246 domains in a read-only fashion otherwise the linux kernels will
247 obviously get very confused as the file system structure may change
248 underneath them (having the same partition mounted rw twice is a sure
249 fire way to cause irreparable damage)! If you want read-write
250 sharing, export the directory to other domains via NFS from domain0.
253 About The Xen Demo CD
254 =====================
256 The purpose of the Demo CD is to distribute a snapshot of Xen's
257 source, and simultaneously provide a convenient means for enabling
258 people to get experience playing with Xen without needing to install
259 it on their hard drive. If you decide to install Xen/XenoLinux you can
260 do so simply by following the installation instructions below -- which
261 essentially involves copying the contents of the CD on to a suitably
262 formated disk partition, and then installing or updating the Grub
263 bootloader.
265 This is a bootable CD that loads Xen, and then a Linux 2.4.22 OS image
266 ported to run on Xen. The CD contains a copy of a file system based on
267 the RedHat 9 distribution that is able to run directly off the CD
268 ("live ISO"), using a "tmpfs" RAM-based file system for root (/etc
269 /var etc). Changes you make to the tmpfs will obviously not be
270 persistent across reboots!
272 Because of the use of a RAM-based file system for root, you'll need
273 plenty of memory to run this CD -- something like 96MB per VM. This is
274 not a restriction of Xen : once you've installed Xen, XenoLinux and
275 the file system images on your hard drive you'll find you can boot VMs
276 in just a few MBs.
278 The CD contains a snapshot of the Xen and XenoLinux code base that we
279 believe to be pretty stable, but lacks some of the features that are
280 currently still work in progress e.g. OS suspend/resume to disk, and
281 various memory management enhancements to provide fast inter-OS
282 communication and sharing of memory pages between OSs. We'll release
283 newer snapshots as required, making use of a BitKeeper repository
284 hosted on http://xen.bkbits.net (follow instructions from the project
285 home page). We're obviously grateful to receive any bug fixes or
286 other code you can contribute. We suggest you join the
287 xen-devel@lists.sourceforge.net mailing list.
290 Installing from the CD
291 ----------------------
293 If you're installing Xen/XenoLinux onto an existing linux file system
294 distribution, just copy the Xen VMM (/boot/image.gz) and XenoLinux
295 kernels (/boot/xenolinux.gz), then modify the Grub config
296 (/boot/grub/menu.lst or /boot/grub/grub.conf) on the target system.
297 It should work on pretty much any distribution.
299 Xen is a "multiboot" standard boot image. Despite being a 'standard',
300 few boot loaders actually support it. The only two we know of are
301 Grub, and our modified version of linux kexec (for booting off a
302 XenoBoot CD -- PlanetLab have adopted the same boot CD approach).
304 If you need to install grub on your system, you can do so either by
305 building the Grub source tree
306 /usr/local/src/grub-0.93-iso9660-splashimage or by copying over all
307 the files in /boot/grub and then running /sbin/grub and following the
308 usual grub documentation. You'll then need to edit the Grub
309 config file.
311 A typical Grub menu option might look like:
313 title Xen / XenoLinux 2.4.22
314 kernel /boot/image.gz dom0_mem=131072 ser_baud=115200 noht
315 module /boot/xenolinux.gz root=/dev/sda4 ro console=tty0
317 The first line specifies which Xen image to use, and what command line
318 arguments to pass to Xen. In this case, we set the maximum amount of
319 memory to allocate to domain0, and the serial baud rate (the default
320 is 9600 baud). We could also disable smp support (nosmp) or disable
321 hyper-threading support (noht). If you have multiple network interface
322 you can use ifname=ethXX to select which one to use. If your network
323 card is unsupported, use ifname=dummy
325 The second line specifies which xenolinux image to use, and the
326 standard linux command line arguments to pass to the kernel. In this
327 case, we're configuring the root partition and stating that it should
328 be mounted read-only (normal practice).
330 If we were booting with an initial ram disk (initrd), then this would
331 require a second "module" line.
334 Installing the file systems from the CD
335 ---------------------------------------
337 If you haven't an existing Linux installation onto which you can just
338 drop down the Xen and XenoLinux images, then the file systems on the
339 CD provide a quick way of doing an install.
341 Choose one or two partitions, depending on whether you want a separate
342 /usr or not. Make file systems on it/them e.g.:
343 mkfs -t ext3 /dev/hda3
344 [or mkfs -t ext2 /dev/hda3 && tune2fs -j /dev/hda3 if using an old
345 version of mkfs]
347 Next, mount the file system(s) e.g.:
348 mkdir /mnt/root && mount /dev/hda3 /mnt/root
349 [mkdir /mnt/usr && mount /dev/hda4 /mnt/usr]
351 To install the root file system, simply untar /usr/XenDemoCD/root.tar.gz:
352 cd /mnt/root && tar -zxpf /usr/XenDemoCD/root.tar.gz
354 You'll need to edit /mnt/root/etc/fstab to reflect your file system
355 configuration. Changing the password file (etc/shadow) is probably a
356 good idea too.
358 To install the usr file system, copy the file system from CD on /usr,
359 though leaving out the "XenDemoCD" and "boot" directories:
360 cd /usr && cp -a X11R6 etc java libexec root src bin dict kerberos local sbin tmp doc include lib man share /mnt/usr
362 If you intend to boot off these file systems (i.e. use them for
363 domain 0), then you probably want to copy the /usr/boot directory on
364 the cd over the top of the current symlink to /boot on your root
365 filesystem (after deleting the current symlink) i.e.:
366 cd /mnt/root ; rm boot ; cp -a /usr/boot .
368 The XenDemoCD directory is only useful if you want to build your own
369 version of the XenDemoCD (see below).
372 Debugging
373 ---------
375 Xen has a set of debugging features that can be useful to try and
376 figure out what's going on. Hit 'h' on the serial line or ScrollLock-h
377 on the keyboard to get a list of supported commands.
379 If you have a crash you'll likely get a crash dump containing an EIP
380 (PC) which, along with an 'objdump -d image', can be useful in
381 figuring out what's happened. Debug a XenoLinux image just as you
382 would any other Linux kernel.
385 Description of how the XenDemoCD boots
386 --------------------------------------
388 1. Grub is used to load Xen, a XenoLinux kernel, and an initrd (initial
389 ram disk). [The source of the version of Grub used is in /usr/local/src]
391 2. the init=/linuxrc command line causes linux to execute /linuxrc in
392 the initrd.
394 3. the /linuxrc file attempts to mount the CD by trying the likely
395 locations : /dev/hd[abcd].
397 4. it then creates a 'tmpfs' file system and untars the
398 'XenDemoCD/root.tar.gz' file into the tmpfs. This contains hopefully
399 all the files that need to be mutable (this would be so much easier
400 if Linux supported 'stacked' or union file systems...)
402 5. Next, /linuxrc uses the pivot_root call to change the root file
403 system to the tmpfs, with the CD mounted as /usr.
405 6. It then invokes /sbin/init in the tmpfs and the boot proceeds
406 normally.
409 Building your own version of the XenDemoCD
410 ------------------------------------------
412 The 'live ISO' version of RedHat is based heavily on Peter Anvin's
413 SuperRescue CD version 2.1.2 and J. McDaniel's Plan-B:
415 http://www.kernel.org/pub/dist/superrescue/v2/
416 http://projectplanb.org/
418 Since Xen uses a "multiboot" image format, it was necessary to change
419 the bootloader from isolinux to Grub0.93 with Leonid Lisovskiy's
420 <lly@pisem.net> grub.0.93-iso9660.patch
422 The Xen Demo CD contains all of the build scripts that were used to
423 create it, so it is possible to 'unpack' the current iso, modifiy it,
424 then build a new iso. The procedure for doing so is as follows:
426 First, mount either the CD, or the iso image of the CD:
428 mount /dev/cdrom /mnt/cdrom
429 or:
430 mount -o loop xendemo-1.0.iso /mnt/cdrom
432 cd to the directory you want to 'unpack' the iso into then run the
433 unpack script:
435 cd /local/xendemocd
436 /mnt/cdrom/XenDemoCD/unpack-iso.sh
438 The result is a 'build' directory containing the file system tree
439 under the 'root' directory. e.g. /local/xendemocd/build/root
441 To add or remove rpms, its possible to use 'rpm' with the --root
442 option to set the path. For more complex changes, it easiest to boot a
443 machine using using the tree via NFS root. Before doing this, you'll
444 need to edit fstab to comment out the seperate mount of /usr.
446 One thing to watch out for: as part of the CD build process, the
447 contents of the 'rootpatch' tree gets copied over the existing 'root'
448 tree replacing various files. The intention of the rootpatch tree is
449 to contain the files that have been modified from the original RH
450 distribution (e.g. various /etc files). This was done to make it
451 easier to upgrade to newer RH versions in the future. The downside of
452 this is that if you edit an existing file in the root tree you should
453 check that you don't also need to propagate the change to the
454 rootpatch tree to avoid it being overwritten.
456 Once you've made the changes and want to build a new iso, here's the
457 procedure:
459 cd /local/xendemocd/build
460 echo '<put_your_name_here>' > Builder
461 ./make.sh put_your_version_id_here >../buildlog 2>&1
463 This process can take 30 mins even on a fast machine, but you should
464 eventually end up with an iso image in the build directory.
466 Notes:
468 root - the root of the file system heirarchy as presented to the
469 running system
471 rootpatch - contains files that have been modified from the standard
472 RH, and copied over the root tree as part of the build
473 procedure.
475 irtree - the file system tree that will go into the initrd (initial
476 ram disk)
478 work - a working directory used in the build process
480 usr - this should really be in 'work' as its created as part of the
481 build process. It contains the 'immutable' files that will
482 be served from the CD rather than the tmpfs containing the
483 contents of root.tar.gz. Some files that are normally in /etc
484 or /var that are large and actually unlikely to need changing
485 have been moved into /usr/root and replaced with links.
488 Ian Pratt
489 9 Sep 2003