view README.CD @ 753:9f659d018e60

bitkeeper revision 1.450.1.1 (3f6cd7e2zhGAHvhWECl313ZSjUy67w)

text change
date Sat Sep 20 22:42:42 2003 +0000 (2003-09-20)
parents 8b83e1b92625
children 893ae00d0c85
line source
1 #############################
2 __ __ _ ___
3 \ \/ /___ _ __ / | / _ \
4 \ // _ \ '_ \ | || | | |
5 / \ __/ | | | | || |_| |
6 /_/\_\___|_| |_| |_(_)___/
8 #############################
10 XenDemoCD 1.0 rc1
11 University of Cambridge Computer Laboratory
12 18 Sep 2003
16 Welcome to the Xen Demo CD!
18 Executive Summary
19 =================
21 This CD is a standalone demo of the Xen Virtual Machine Monitor (VMM)
22 and Linux-2.4 OS port (XenoLinux). It runs entirely off the CD,
23 without requiring hard disk installation. This is achieved using a RAM
24 disk to store mutable file system data while using the CD for
25 everything else. The CD can also be used for installing Xen/XenoLinux
26 to disk, and includes a source code snapshot along with all of the
27 tools required to build it.
29 Booting the CD
30 ==============
32 The Xen VMM is currently fairly h/w specific, but porting new device
33 drivers is relatively straightforward thanks to Xen's Linux driver
34 compatibility layer. The current snapshot supports the following
35 hardware:
37 CPU: Pentium Pro/II/III/IV/Xeon, Athlon (i.e. P6 or newer) SMP supported
38 IDE: Intel PIIX chipset, others will be PIO only (slow)
39 SCSI: Adaptec / Dell PERC Raid (aacraid), megaraid, Adaptec aic7xxx
40 Net: Recommended: Intel e1000, Broadcom BCM57xx (tg3), 3c905 (3c59x)
41 Tested but require extra copies : pcnet32
42 Untested and also requires extra copies : Intel e100, tulip
44 Because of the demo CD's use of RAM disks, make sure you have plenty
45 of RAM (256MB+).
47 To try out the Demo, boot from CD (you may need to change your BIOS
48 configuration to do this), hit a key on either the keyboard or serial
49 line to pull up the Grub boot menu, then select one of the four boot
50 options:
52 Xen / linux-2.4.22
53 Xen / linux-2.4.22 using cmdline IP configuration
54 Xen ? linux-2.4.22 in "safe mode"
55 linux-2.4.22
57 The last option is a plain linux kernel that runs on the bare machine,
58 and is included simply to help diagnose driver compatibility
59 problems. The "safe mode" boot option might be useful if you're having
60 problems getting Xen to work with your hardware, as it disables various
61 features such as SMP, and enables some debugging.
63 If you are going for a command line IP config, hit "e" at
64 the grub menu, then edit the "ip=" parameters to reflect your setup
65 e.g. "ip=<ipaddr>::<gateway>:<netmask>::eth0:off". It shouldn't be
66 necessary to set either the nfs server or hostname
67 parameters. Alternatively, once XenoLinux has booted you can login and
68 setup networking with 'dhclient' or 'ifconfig' and 'route' in the
69 normal way.
71 To make things easier for yourself, it's worth trying to arrange for an
72 IP address which is the first in a sequential range of free IP
73 addresses. It's useful to give each VM instance its own public IP
74 address (though it is possible to do NAT or use private addresses),
75 and the configuration files on the CD allocate IP addresses
76 sequentially for subsequent domains unless told otherwise.
78 After selecting the kernel to boot, stand back and watch Xen boot,
79 closely followed by "domain 0" running the XenoLinux kernel. The boot
80 messages are also sent to the serial line (the baud rate can be set on
81 the Xen cmdline, but defaults to 115200), which can be very useful for
82 debugging should anything important scroll off the screen. Xen's
83 startup messages will look quite familiar as much of the hardware
84 initialisation (SMP boot, apic setup) and device drivers are derived
85 from Linux.
87 If everything is well, you should see the linux rc scripts start a
88 bunch of standard services including sshd. Login on the console or
89 via ssh::
90 username: user root
91 password: xendemo xendemo
93 Once logged in, it should look just like any regular linux box. All
94 the usual tools and commands should work as per usual. It's probably
95 best to start by configuring networking, either with 'dhclient' or
96 manually via ifconfig and route, remembering to edit /etc/resolv.conf
97 if you want DNS.
99 You can start an X server with 'startx'. It defaults to a conservative
100 1024x768, but you can edit the script for higher resoloutions. The CD
101 contains a load of standard software. You should be able to start
102 Apache, PostgreSQL, Mozzila etc in the normal way, but because
103 everything is running off CD the performance will be very sluggish and
104 you may run out of memory for the 'tmpfs' file system. You may wish
105 to go ahead and install Xen/XenoLinux on your hard drive, either
106 dropping Xen and the XenoLinux kernel down onto a pre-existing Linux
107 distribution, or using the file systems from the CD (which are based
108 on RH9). See the installation instructions later in this document.
110 If you want to browse the Xen / XenoLinux source, it's all located
111 under /usr/local/src, complete with BitKeeper repository. We've also
112 included source code and configuration information for the various
113 benchmarks we used in the SOSP paper.
116 Starting other domains
117 ======================
119 There's a web interface for starting and managing other domains (VMs),
120 but since we generally use the command line tools they're probably
121 rather better debugged at present. The key command is 'xenctl' which
122 lives in /usr/local/bin and uses /etc/xenctl.xml for its default
123 configuration. Run 'xenctl' without any arguments to get a help
124 message. Note that xenctl is a java front end to various underlying
125 internal tools written in C (xi_*). Running off CD, it seems to take
126 an age to start...
128 Abyway, the first thing to do is to set up a window in which you will
129 receive console output from other domains. Console output will arrive
130 as UDP packets destined for, so its necessary to setup an
131 alias on eth0. The easiest way to do this is to run:
133 xen_nat_enable
135 This also inserts a few NAT rules into "domain0", in case you'll be
136 starting other domains without their own IP addresses. Alternatively,
137 just do "ifconfig eth0:0 up". NB: The intention is that in
138 future Xen will do NAT itsel (actually RSIP), but this is part of a
139 larger work package that isn't stable enough to release.
141 Next, run a the xen UDP console displayer:
143 xen_read_console &
146 As mentioned above, xenctl uses /etc/xenctl.xml as its default
147 configuration. The directory contains two different configs depending
148 on whether you want to use NAT, or multiple sequential external IPs
149 (it's possible to override any of the parameters on the command line
150 if you want to set specific IPs, etc).
152 The default configuration file supports NAT. To change to use multiple IPs:
154 cp /etc/xenctl.xml-publicip /etc/xenctl.xml
156 A sequence of commands must be given to xenctl to start a new
157 domain. First a new domain must be created, which requires specifying
158 the initial memory allocation, the kernel image to use, and the kernel
159 command line. As well as the root file system details, you'll need to
160 set the IP address on the command line: since Xen currently doesn't
161 support a virtual console for domains >1, you won't be able to log to
162 your new domain unless you've got networking configured and an sshd
163 running! (using dhcp for new domains should work too).
165 After creating the domain, xenctl must be used to grant the domain
166 access to other resources such as physical or virtual disk partions.
167 Then, the domain must be started.
169 These commands can be entered manually, but for convenience, xenctl
170 will also read them from a script and infer which domain number you're
171 referring to (-nX). To use the sample script:
173 xenctl script -f/etc/xen-mynewdom [NB: no space after the -f]
175 You should see the domain booting on your xen_read_console window.
177 The xml defaults start another domain running off the CD, using a
178 separate RAM-based file system for mutable data in root (just like
179 domain 0).
181 The new domain is started with a '4' on the kernel command line to
182 tell 'init' to go to runlevel 4 rather than the default of 3. This is
183 done simply to suppress a bunch of harmless error messages that would
184 otherwise occur when the new (unprivileged) domain tried to access
185 physical hardware resources to try setting the hwclock, system font,
186 run gpm etc.
188 After it's booted, you should be able to ssh into your new domain. If
189 you went for a NATed address, from domain 0 you should be able to ssh
190 into '169.254.1.X' where X is the domain number. If you ran the
191 xen_enable_nat script, a bunch of port redirects have been installed
192 to enable you to ssh in to other domains remotely. To access the new
193 virtual machine remotely, use:
195 ssh -p2201 root@IP.address.Of.Domain0 # use 2202 for domain 2 etc.
197 If you configured the new domain with its own IP address, you should
198 be able to ssh into it directly.
201 "xenctl domain list" provides status information about running domains,
202 though is currently only allowed to be run by domain 0. It accesses
203 /proc/xeno/domains to read this information from Xen. You can also use
204 xenctl to 'stop' (pause) a domain, or 'kill' a domain. You can either
205 kill it nicely by sending a shutdown event and waiting for it to
206 terminate, or blow the sucker away with extreme prejudice.
208 If you want to configure a new domain differently, type 'xenctl' to
209 get a list of arguments, e.g. at the 'xenctl domain new' command line
210 use the "-4" option to set a diffrent IPv4 address.
212 xenctl can be used to set the new kernel's command line, and hence
213 determine what it uses as a root file system, etc. Although the default
214 is to boot in the same manner that domain0 did (using the RAM-based
215 file system for root and the CD for /usr) it's possible to configure any
216 of the following possibilities, for example:
218 * initrd=/boot/initrd init=/linuxrc
219 boot using an initial ram disk, executing /linuxrc (as per this CD)
221 * root=/dev/hda3 ro
222 boot using a standard hard disk partition as root
224 * root=/dev/xvda1 ro
225 boot using a pre-configured 'virtual block device' that will be
226 attached to a virtual disk that previously has had a file system
227 installed on it.
229 * root=/dev/nfs nfsroot=/path/on/server ip=<blah_including server_IP>
230 Boot using an NFS mounted root file system. This could be from a
231 remote NFS server, or from an NFS server running in another
232 domain. The latter is rather a useful option.
235 A typical setup might be to allocate a standard disk partition for
236 each domain and populate it with files. To save space, having a shared
237 read-only usr partition might make sense.
239 Alternatively, you can use 'virtual disks', which are stored as files
240 within a custom file system. "xenctl partitions add" can be used to
241 'format' a partition with the file system, and then virtual disks can
242 be created with "xenctl vd create". Virtual disks can then be attached
243 to a running domain as a 'virtual block device' using "xenctl vdb
244 create". The virtual disk can optionally be partitioned (e.g. "fdisk
245 /dev/xvda") or have a file system created on it directly (e.g. "mkfs
246 -t ext3 /dev/xvda"). The virtual disk can then be accessed by a
247 virtual block device associated with another domain, and even used as
248 a boot device.
250 Both virtual disks and real partitions should only be shared between
251 domains in a read-only fashion otherwise the linux kernels will
252 obviously get very confused as the file system structure may change
253 underneath them (having the same partition mounted rw twice is a sure
254 fire way to cause irreparable damage)! If you want read-write
255 sharing, export the directory to other domains via NFS from domain0.
258 Troubleshooting Problems
259 ========================
261 If you have problems booting Xen, there are a number of boot parameters
262 that may be able to help diagnose problems:
264 ignorebiostables Disable parsing of BIOS-supplied tables. This is needed
265 for some very unsupported chipsets (eg. nforce2). If you
266 specify this option then ACPI tables are also ignored, and
267 SMP suppirt is disabled.
269 nosmp Disable SMP support.
270 This option is implied by 'ignorebiostables'.
272 noacpi Disable ACPI tables, which confuse Xen on some chipsets.
273 This option is implied by 'ignorebiostables'.
275 watchdog Enable NMI watchdog which can report certain failures.
277 noht Disable Hyperthreading.
279 ifname=ethXX Select which Ethernet interface to use.
281 ifname=dummy Don't use any network interface.
283 ser_baud=xxx Set serial line baud rate for console.
285 dom0_mem=xxx Set the initial amount of memory for domain0.
288 It's probably a good idea to join the Xen developer's mailing list on
289 Sourceforge:
292 About The Xen Demo CD
293 =====================
295 The purpose of the Demo CD is to distribute a snapshot of Xen's
296 source, and simultaneously provide a convenient means for enabling
297 people to get experience playing with Xen without needing to install
298 it on their hard drive. If you decide to install Xen/XenoLinux you can
299 do so simply by following the installation instructions below -- which
300 essentially involves copying the contents of the CD on to a suitably
301 formated disk partition, and then installing or updating the Grub
302 bootloader.
304 This is a bootable CD that loads Xen, and then a Linux 2.4.22 OS image
305 ported to run on Xen. The CD contains a copy of a file system based on
306 the RedHat 9 distribution that is able to run directly off the CD
307 ("live ISO"), using a "tmpfs" RAM-based file system for root (/etc
308 /var etc). Changes you make to the tmpfs will obviously not be
309 persistent across reboots!
311 Because of the use of a RAM-based file system for root, you'll need
312 plenty of memory to run this CD -- something like 96MB per VM. This is
313 not a restriction of Xen : once you've installed Xen, XenoLinux and
314 the file system images on your hard drive you'll find you can boot VMs
315 in just a few MBs.
317 The CD contains a snapshot of the Xen and XenoLinux code base that we
318 believe to be pretty stable, but lacks some of the features that are
319 currently still work in progress e.g. OS suspend/resume to disk, and
320 various memory management enhancements to provide fast inter-OS
321 communication and sharing of memory pages between OSs. We'll release
322 newer snapshots as required, making use of a BitKeeper repository
323 hosted on (follow instructions from the project
324 home page). We're obviously grateful to receive any bug fixes or
325 other code you can contribute. We suggest you join the
326 mailing list.
329 Installing from the CD
330 ----------------------
332 If you're installing Xen/XenoLinux onto an existing linux file system
333 distribution, just copy the Xen VMM (/boot/image.gz) and XenoLinux
334 kernels (/boot/xenolinux.gz), then modify the Grub config
335 (/boot/grub/menu.lst or /boot/grub/grub.conf) on the target system.
336 It should work on pretty much any distribution.
338 Xen is a "multiboot" standard boot image. Despite being a 'standard',
339 few boot loaders actually support it. The only two we know of are
340 Grub, and our modified version of linux kexec (for booting off a
341 XenoBoot CD -- PlanetLab have adopted the same boot CD approach).
343 If you need to install grub on your system, you can do so either by
344 building the Grub source tree
345 /usr/local/src/grub-0.93-iso9660-splashimage or by copying over all
346 the files in /boot/grub and then running /sbin/grub and following the
347 usual grub documentation. You'll then need to edit the Grub
348 config file.
350 A typical Grub menu option might look like:
352 title Xen / XenoLinux 2.4.22
353 kernel /boot/image.gz dom0_mem=131072 ser_baud=115200 noht
354 module /boot/xenolinux.gz root=/dev/sda4 ro console=tty0
356 The first line specifies which Xen image to use, and what command line
357 arguments to pass to Xen. In this case, we set the maximum amount of
358 memory to allocate to domain0, and the serial baud rate (the default
359 is 9600 baud). We could also disable smp support (nosmp) or disable
360 hyper-threading support (noht). If you have multiple network interface
361 you can use ifname=ethXX to select which one to use. If your network
362 card is unsupported, use ifname=dummy
364 The second line specifies which xenolinux image to use, and the
365 standard linux command line arguments to pass to the kernel. In this
366 case, we're configuring the root partition and stating that it should
367 be mounted read-only (normal practice).
369 If we were booting with an initial ram disk (initrd), then this would
370 require a second "module" line.
373 Installing the file systems from the CD
374 ---------------------------------------
376 If you haven't an existing Linux installation onto which you can just
377 drop down the Xen and XenoLinux images, then the file systems on the
378 CD provide a quick way of doing an install.
380 Choose one or two partitions, depending on whether you want a separate
381 /usr or not. Make file systems on it/them e.g.:
382 mkfs -t ext3 /dev/hda3
383 [or mkfs -t ext2 /dev/hda3 && tune2fs -j /dev/hda3 if using an old
384 version of mkfs]
386 Next, mount the file system(s) e.g.:
387 mkdir /mnt/root && mount /dev/hda3 /mnt/root
388 [mkdir /mnt/usr && mount /dev/hda4 /mnt/usr]
390 To install the root file system, simply untar /usr/XenDemoCD/root.tar.gz:
391 cd /mnt/root && tar -zxpf /usr/XenDemoCD/root.tar.gz
393 You'll need to edit /mnt/root/etc/fstab to reflect your file system
394 configuration. Changing the password file (etc/shadow) is probably a
395 good idea too.
397 To install the usr file system, copy the file system from CD on /usr,
398 though leaving out the "XenDemoCD" and "boot" directories:
399 cd /usr && cp -a X11R6 etc java libexec root src bin dict kerberos local sbin tmp doc include lib man share /mnt/usr
401 If you intend to boot off these file systems (i.e. use them for
402 domain 0), then you probably want to copy the /usr/boot directory on
403 the cd over the top of the current symlink to /boot on your root
404 filesystem (after deleting the current symlink) i.e.:
405 cd /mnt/root ; rm boot ; cp -a /usr/boot .
407 The XenDemoCD directory is only useful if you want to build your own
408 version of the XenDemoCD (see below).
411 Debugging
412 ---------
414 Xen has a set of debugging features that can be useful to try and
415 figure out what's going on. Hit 'h' on the serial line or ScrollLock-h
416 on the keyboard to get a list of supported commands.
418 If you have a crash you'll likely get a crash dump containing an EIP
419 (PC) which, along with an 'objdump -d image', can be useful in
420 figuring out what's happened. Debug a XenoLinux image just as you
421 would any other Linux kernel.
424 Description of how the XenDemoCD boots
425 --------------------------------------
427 1. Grub is used to load Xen, a XenoLinux kernel, and an initrd (initial
428 ram disk). [The source of the version of Grub used is in /usr/local/src]
430 2. the init=/linuxrc command line causes linux to execute /linuxrc in
431 the initrd.
433 3. the /linuxrc file attempts to mount the CD by trying the likely
434 locations : /dev/hd[abcd].
436 4. it then creates a 'tmpfs' file system and untars the
437 'XenDemoCD/root.tar.gz' file into the tmpfs. This contains hopefully
438 all the files that need to be mutable (this would be so much easier
439 if Linux supported 'stacked' or union file systems...)
441 5. Next, /linuxrc uses the pivot_root call to change the root file
442 system to the tmpfs, with the CD mounted as /usr.
444 6. It then invokes /sbin/init in the tmpfs and the boot proceeds
445 normally.
448 Building your own version of the XenDemoCD
449 ------------------------------------------
451 The 'live ISO' version of RedHat is based heavily on Peter Anvin's
452 SuperRescue CD version 2.1.2 and J. McDaniel's Plan-B:
457 Since Xen uses a "multiboot" image format, it was necessary to change
458 the bootloader from isolinux to Grub0.93 with Leonid Lisovskiy's
459 <> grub.0.93-iso9660.patch
461 The Xen Demo CD contains all of the build scripts that were used to
462 create it, so it is possible to 'unpack' the current iso, modifiy it,
463 then build a new iso. The procedure for doing so is as follows:
465 First, mount either the CD, or the iso image of the CD:
467 mount /dev/cdrom /mnt/cdrom
468 or:
469 mount -o loop xendemo-1.0.iso /mnt/cdrom
471 cd to the directory you want to 'unpack' the iso into then run the
472 unpack script:
474 cd /local/xendemocd
475 /mnt/cdrom/XenDemoCD/
477 The result is a 'build' directory containing the file system tree
478 under the 'root' directory. e.g. /local/xendemocd/build/root
480 To add or remove rpms, its possible to use 'rpm' with the --root
481 option to set the path. For more complex changes, it easiest to boot a
482 machine using using the tree via NFS root. Before doing this, you'll
483 need to edit fstab to comment out the seperate mount of /usr.
485 One thing to watch out for: as part of the CD build process, the
486 contents of the 'rootpatch' tree gets copied over the existing 'root'
487 tree replacing various files. The intention of the rootpatch tree is
488 to contain the files that have been modified from the original RH
489 distribution (e.g. various /etc files). This was done to make it
490 easier to upgrade to newer RH versions in the future. The downside of
491 this is that if you edit an existing file in the root tree you should
492 check that you don't also need to propagate the change to the
493 rootpatch tree to avoid it being overwritten.
495 Once you've made the changes and want to build a new iso, here's the
496 procedure:
498 cd /local/xendemocd/build
499 echo '<put_your_name_here>' > Builder
500 ./ put_your_version_id_here >../buildlog 2>&1
502 This process can take 30 mins even on a fast machine, but you should
503 eventually end up with an iso image in the build directory.
505 Notes:
507 root - the root of the file system heirarchy as presented to the
508 running system
510 rootpatch - contains files that have been modified from the standard
511 RH, and copied over the root tree as part of the build
512 procedure.
514 irtree - the file system tree that will go into the initrd (initial
515 ram disk)
517 work - a working directory used in the build process
519 usr - this should really be in 'work' as its created as part of the
520 build process. It contains the 'immutable' files that will
521 be served from the CD rather than the tmpfs containing the
522 contents of root.tar.gz. Some files that are normally in /etc
523 or /var that are large and actually unlikely to need changing
524 have been moved into /usr/root and replaced with links.
527 Ian Pratt
528 9 Sep 2003