view README.CD @ 1041:7c3b17193b4b

bitkeeper revision 1.680 (40083bb4Kb4q4cb9z5eySYtCRnVh3A)

author kaf24@scramble.cl.cam.ac.uk[kaf24]
date Fri Jan 16 19:29:56 2004 +0000 (2004-01-16)
parents 6ec887aa9d16
children 9ccd60698e09 5fc679c3c810
line source
1 #############################
2 __ __ _ ___
3 \ \/ /___ _ __ / | / _ \
4 \ // _ \ '_ \ | || | | |
5 / \ __/ | | | | || |_| |
6 /_/\_\___|_| |_| |_(_)___/
8 #############################
10 XenDemoCD 1.0
11 University of Cambridge Computer Laboratory
12 29 Sep 2003
14 http://www.cl.cam.ac.uk/netos/xen
16 Welcome to the Xen Demo CD!
18 Executive Summary
19 =================
21 This CD is a standalone demo of the Xen Virtual Machine Monitor (VMM)
22 and Linux-2.4 OS port (XenoLinux). It runs entirely off the CD,
23 without requiring hard disk installation. This is achieved using a RAM
24 disk to store mutable file system data while using the CD for
25 everything else. The CD can also be used for installing Xen/XenoLinux
26 to disk, and includes a source code snapshot along with all of the
27 tools required to build it.
29 Booting the CD
30 ==============
32 The Xen VMM is currently fairly h/w specific, but porting new device
33 drivers is relatively straightforward thanks to Xen's Linux driver
34 compatibility layer. The current snapshot supports the following
35 hardware:
37 CPU: Pentium Pro/II/III/IV/Xeon, Athlon (i.e. P6 or newer) SMP supported
38 IDE: Intel PIIX chipset, others will be PIO only (slow)
39 SCSI: Adaptec / Dell PERC Raid (aacraid), megaraid, Adaptec aic7xxx
40 Net: Recommended: Intel e1000, Broadcom BCM57xx (tg3), 3c905 (3c59x)
41 Working, but require extra copies : pcnet32, Intel e100, tulip
43 Because of the demo CD's use of RAM disks, make sure you have plenty
44 of RAM (256MB+).
46 To try out the Demo, boot from CD (you may need to change your BIOS
47 configuration to do this), then select one of the four boot options
48 from the Grub menu:
50 Xen / linux-2.4.22
51 Xen / linux-2.4.22 using cmdline IP configuration
52 Xen / linux-2.4.22 in "safe mode"
53 linux-2.4.22
55 The last option is a plain linux kernel that runs on the bare machine,
56 and is included simply to help diagnose driver compatibility
57 problems. The "safe mode" boot option might be useful if you're having
58 problems getting Xen to work with your hardware, as it disables various
59 features such as SMP, and enables some debugging.
61 If you are going for a command line IP config, hit "e" at
62 the grub menu, then edit the "ip=" parameters to reflect your setup
63 e.g. "ip=<ipaddr>::<gateway>:<netmask>::eth0:off". It shouldn't be
64 necessary to set either the nfs server or hostname
65 parameters. Alternatively, once XenoLinux has booted you can login and
66 setup networking with 'dhclient' or 'ifconfig' and 'route' in the
67 normal way.
69 To make things easier for yourself, it's worth trying to arrange for an
70 IP address which is the first in a sequential range of free IP
71 addresses. It's useful to give each VM instance its own public IP
72 address (though it is possible to do NAT or use private addresses),
73 and the configuration files on the CD allocate IP addresses
74 sequentially for subsequent domains unless told otherwise.
76 After selecting the kernel to boot, stand back and watch Xen boot,
77 closely followed by "domain 0" running the XenoLinux kernel. The boot
78 messages can also sent to the serial line by specifying the baud rate
79 on the Xen cmdline (e.g., 'ser_baud=9600'); this can be very useful
80 for debugging should anything important scroll off the screen. Xen's
81 startup messages will look quite familiar as much of the hardware
82 initialisation (SMP boot, apic setup) and device drivers are derived
83 from Linux.
85 If everything is well, you should see the linux rc scripts start a
86 bunch of standard services including sshd. Login on the console or
87 via ssh::
88 username: user root
89 password: xendemo xendemo
91 Once logged in, it should look just like any regular linux box. All
92 the usual tools and commands should work as per usual. However,
93 because of the poor random access performance of CD drives, the
94 machine will feel rather slugish, and you may run out of memory if you
95 make significant modifications to the ramfs filesystem -- for the full
96 experience, install a Xen and XenoLinux image on you hard drive :-)
98 You can configure networking, either with 'dhclient' or manually via
99 'ifconfig' and 'route', remembering to edit /etc/resolv.conf if you
100 want DNS.
102 You can start an X server with 'startx'. It defaults to a conservative
103 1024x768, but you can edit the script for higher resoloutions. The CD
104 contains a load of standard software. You should be able to start
105 Apache, PostgreSQL, Mozilla etc in the normal way, but because
106 everything is running off CD the performance will be very sluggish and
107 you may run out of memory for the 'tmpfs' file system. You may wish
108 to go ahead and install Xen/XenoLinux on your hard drive, either
109 dropping Xen and the XenoLinux kernel down onto a pre-existing Linux
110 distribution, or using the file systems from the CD (which are based
111 on RH9). See the installation instructions later in this document.
113 If your video card requires 'agpgart' then it unfortunately won't yet
114 work with Xen, and you'll only be able to configure a VGA X
115 server. We're working on a fix for this for the next release.
117 If you want to browse the Xen / XenoLinux source, it's all located
118 under /usr/local/src, complete with BitKeeper repository. We've also
119 included source code and configuration information for the various
120 benchmarks we used in the SOSP paper.
123 Starting other domains
124 ======================
126 Xen's privileged control interfaces can be accessed using a handy C
127 library (libxc.so) or an even easier-to-use Python wrapper module
128 (Xc). Example script templates are provided in tools/examples/.
130 Abyway, the first thing to do is to set up a window in which you will
131 receive console output from other domains. Console output will arrive
132 as UDP packets destined for, so its necessary to setup an
133 alias on eth0. The easiest way to do this is to run:
135 xen_nat_enable
137 This also inserts a few NAT rules into "domain0", in case you'll be
138 starting other domains without their own IP addresses. Alternatively,
139 just do "ifconfig eth0:0 up". NB: The intention is that in
140 future Xen will do NAT itsel (actually RSIP), but this is part of a
141 larger work package that isn't stable enough to release.
143 Next, run a the xen UDP console displayer:
145 xen_read_console &
148 As mentioned above, a template Python script is provided:
149 tools/examples/craetelinuxdom.py. This can be modified to set up a
150 virtual Ethernet interface, access to local discs, and various other
151 parameters.
153 When you execute your modified screipt, you should see the domain
154 booting on your xen_read_console window.
156 The new domain may be started with a '4' on the kernel command line to
157 tell 'init' to go to runlevel 4 rather than the default of 3. This is
158 done simply to suppress a bunch of harmless error messages that would
159 otherwise occur when the new (unprivileged) domain tried to access
160 physical hardware resources to try setting the hwclock, system font,
161 run gpm etc.
163 After it's booted, you should be able to ssh into your new domain. If
164 you went for a NATed address, from domain 0 you should be able to ssh
165 into '169.254.1.X' where X is the domain number. If you ran the
166 xen_enable_nat script, a bunch of port redirects have been installed
167 to enable you to ssh in to other domains remotely. To access the new
168 virtual machine remotely, use:
170 ssh -p2201 root@IP.address.Of.Domain0 # use 2202 for domain 2 etc.
172 If you configured the new domain with its own IP address, you should
173 be able to ssh into it directly.
175 Script 'tools/examples/listdoms.py' demonstrates how to generate a
176 list of all extant domains. Prettier printing is an exercise for the
177 reader!
179 createlinuxdom.py can be used to set the new kernel's command line,
180 and hence determine what it uses as a root file system, etc. Although
181 the default is to boot in the same manner that domain0 did (using the
182 RAM-based file system for root and the CD for /usr) it's possible to
183 configure any of the following possibilities, for example:
185 * initrd=/boot/initrd init=/linuxrc
186 boot using an initial ram disk, executing /linuxrc (as per this CD)
188 * root=/dev/hda3 ro
189 boot using a standard hard disk partition as root
190 !!! remember to grant access in createlinuxdom.py.
192 * root=/dev/xvda1 ro
193 boot using a pre-configured 'virtual block device' that will be
194 attached to a virtual disk that previously has had a file system
195 installed on it.
197 * root=/dev/nfs nfsroot=/path/on/server ip=<blah_including server_IP>
198 Boot using an NFS mounted root file system. This could be from a
199 remote NFS server, or from an NFS server running in another
200 domain. The latter is rather a useful option.
202 A typical setup might be to allocate a standard disk partition for
203 each domain and populate it with files. To save space, having a shared
204 read-only usr partition might make sense.
206 Block devices should only be shared between domains in a read-only
207 fashion otherwise the linux kernels will obviously get very confused
208 as the file system structure may change underneath them (having the
209 same partition mounted rw twice is a sure fire way to cause
210 irreparable damage)! If you want read-write sharing, export the
211 directory to other domains via NFS from domain0.
214 Troubleshooting Problems
215 ========================
217 If you have problems booting Xen, there are a number of boot parameters
218 that may be able to help diagnose problems:
220 ignorebiostables Disable parsing of BIOS-supplied tables. This may
221 help with some chipsets that aren't fully supported
222 by Xen. If you specify this option then ACPI tables are
223 also ignored, and SMP support is disabled.
225 noreboot Don't reboot the machine automatically on errors.
226 This is useful to catch debug output if you aren't
227 catching console messages via the serial line.
229 nosmp Disable SMP support.
230 This option is implied by 'ignorebiostables'.
232 noacpi Disable ACPI tables, which confuse Xen on some chipsets.
233 This option is implied by 'ignorebiostables'.
235 watchdog Enable NMI watchdog which can report certain failures.
237 noht Disable Hyperthreading.
239 ifname=ethXX Select which Ethernet interface to use.
241 ifname=dummy Don't use any network interface.
243 ser_baud=xxx Enable serial I/O and set the baud rate.
245 dom0_mem=xxx Set the initial amount of memory for domain0.
248 It's probably a good idea to join the Xen developer's mailing list on
249 Sourceforge: http://lists.sourceforge.net/lists/listinfo/xen-devel
252 About The Xen Demo CD
253 =====================
255 The purpose of the Demo CD is to distribute a snapshot of Xen's
256 source, and simultaneously provide a convenient means for enabling
257 people to get experience playing with Xen without needing to install
258 it on their hard drive. If you decide to install Xen/XenoLinux you can
259 do so simply by following the installation instructions below -- which
260 essentially involves copying the contents of the CD on to a suitably
261 formated disk partition, and then installing or updating the Grub
262 bootloader.
264 This is a bootable CD that loads Xen, and then a Linux 2.4.22 OS image
265 ported to run on Xen. The CD contains a copy of a file system based on
266 the RedHat 9 distribution that is able to run directly off the CD
267 ("live ISO"), using a "tmpfs" RAM-based file system for root (/etc
268 /var etc). Changes you make to the tmpfs will obviously not be
269 persistent across reboots!
271 Because of the use of a RAM-based file system for root, you'll need
272 plenty of memory to run this CD -- something like 96MB per VM. This is
273 not a restriction of Xen : once you've installed Xen, XenoLinux and
274 the file system images on your hard drive you'll find you can boot VMs
275 in just a few MBs.
277 The CD contains a snapshot of the Xen and XenoLinux code base that we
278 believe to be pretty stable, but lacks some of the features that are
279 currently still work in progress e.g. OS suspend/resume to disk, and
280 various memory management enhancements to provide fast inter-OS
281 communication and sharing of memory pages between OSs. We'll release
282 newer snapshots as required, making use of a BitKeeper repository
283 hosted on http://xen.bkbits.net (follow instructions from the project
284 home page). We're obviously grateful to receive any bug fixes or
285 other code you can contribute. We suggest you join the
286 xen-devel@lists.sourceforge.net mailing list.
289 Installing from the CD
290 ======================
292 If you're installing Xen/XenoLinux onto an existing linux file system
293 distribution, just copy the Xen VMM (/boot/image.gz) and XenoLinux
294 kernels (/boot/xenolinux.gz), then modify the Grub config
295 (/boot/grub/menu.lst or /boot/grub/grub.conf) on the target system.
296 It should work on pretty much any distribution.
298 Xen is a "multiboot" standard boot image. Despite being a 'standard',
299 few boot loaders actually support it. The only two we know of are
300 Grub, and our modified version of linux kexec (for booting off a
301 XenoBoot CD -- PlanetLab have adopted the same boot CD approach).
303 If you need to install grub on your system, you can do so either by
304 building the Grub source tree
305 /usr/local/src/grub-0.93-iso9660-splashimage or by copying over all
306 the files in /boot/grub and then running /sbin/grub and following the
307 usual grub documentation. You'll then need to edit the Grub
308 config file.
310 A typical Grub menu option might look like:
312 title Xen / XenoLinux 2.4.22
313 kernel /boot/image.gz dom0_mem=131072 ser_baud=115200 noht
314 module /boot/xenolinux.gz root=/dev/sda4 ro console=tty0
316 The first line specifies which Xen image to use, and what command line
317 arguments to pass to Xen. In this case we set the maximum amount of
318 memory to allocate to domain0, and enable serial I/O at 9600 baud.
319 We could also disable smp support (nosmp) or disable hyper-threading
320 support (noht). If you have multiple network interface you can use
321 ifname=ethXX to select which one to use. If your network card is
322 unsupported, use ifname=dummy
324 The second line specifies which xenolinux image to use, and the
325 standard linux command line arguments to pass to the kernel. In this
326 case, we're configuring the root partition and stating that it should
327 be mounted read-only (normal practice).
329 If we were booting with an initial ram disk (initrd), then this would
330 require a second "module" line.
333 Installing the file systems from the CD
334 =======================================
336 If you haven't an existing Linux installation onto which you can just
337 drop down the Xen and XenoLinux images, then the file systems on the
338 CD provide a quick way of doing an install.
340 Choose one or two partitions, depending on whether you want a separate
341 /usr or not. Make file systems on it/them e.g.:
342 mkfs -t ext3 /dev/hda3
343 [or mkfs -t ext2 /dev/hda3 && tune2fs -j /dev/hda3 if using an old
344 version of mkfs]
346 Next, mount the file system(s) e.g.:
347 mkdir /mnt/root && mount /dev/hda3 /mnt/root
348 [mkdir /mnt/usr && mount /dev/hda4 /mnt/usr]
350 To install the root file system, simply untar /usr/XenDemoCD/root.tar.gz:
351 cd /mnt/root && tar -zxpf /usr/XenDemoCD/root.tar.gz
353 You'll need to edit /mnt/root/etc/fstab to reflect your file system
354 configuration. Changing the password file (etc/shadow) is probably a
355 good idea too.
357 To install the usr file system, copy the file system from CD on /usr,
358 though leaving out the "XenDemoCD" and "boot" directories:
359 cd /usr && cp -a X11R6 etc java libexec root src bin dict kerberos local sbin tmp doc include lib man share /mnt/usr
361 If you intend to boot off these file systems (i.e. use them for
362 domain 0), then you probably want to copy the /usr/boot directory on
363 the cd over the top of the current symlink to /boot on your root
364 filesystem (after deleting the current symlink) i.e.:
365 cd /mnt/root ; rm boot ; cp -a /usr/boot .
367 The XenDemoCD directory is only useful if you want to build your own
368 version of the XenDemoCD (see below).
371 Debugging
372 =========
374 Xen has a set of debugging features that can be useful to try and
375 figure out what's going on. Hit 'h' on the serial line (if you
376 specified a baud rate on the Xen command line) or ScrollLock-h on the
377 keyboard to get a list of supported commands.
379 If you have a crash you'll likely get a crash dump containing an EIP
380 (PC) which, along with an 'objdump -d image', can be useful in
381 figuring out what's happened. Debug a XenoLinux image just as you
382 would any other Linux kernel.
384 We supply a handy debug terminal program which you can find in
385 /usr/local/src/xen-1.0/xeno-1.0.bk/tools/misc/miniterm/
386 This should be built and executed on another machine that is connected
387 via a null modem cable. Documentation is included.
388 Alternatively, telnet can be used in 'char mode' if the Xen machine is
389 connected to a serial-port server.
392 Installing Xen / XenoLinux on a RedHat distribution
393 ===================================================
395 When using Xen / Xenolinux on a standard Linux distribution there are
396 a couple of things to watch out for:
398 The first Linux VM that is started when Xen boots start (Domain 0) is
399 given direct access to the graphics card, so it may use it as a
400 console. Other domains don't have ttyN consoles, so attempts to run a
401 'mingetty' against them will fail, generating periodic warning
402 messages from 'init' about services respawning too fast. They should
403 work for domain0 just fine.
405 In future, we may make the current 'xencons' accept input as well as
406 output, so that a getty can be run against it. In the meantime, other
407 domains don't have a console suitable for logging in on, so you'll
408 have to run sshd and ssh in to them.
410 To prevent the warning messages you'll need to remove them from
411 /etc/inittab for domains>0. Due to a bug in the RH9 /etc/rc.sysinit
412 script #'ing the lines out of /etc/inittab won't work as it ignores
413 the '#' and tries to access them anyway.
415 Also, because domains>0 don't have any privileged access at all,
416 certain commands in the default boot sequence will fail e.g. attempts
417 to update the hwclock, change the console font, update the keytable
418 map, start apmd (power management), or gpm (mouse cursor). Either
419 ignore the errors, or remove them from the startup scripts. Deleting
420 the following links are a good start: S24pcmcia S09isdn S17keytable
421 S26apmd S85gpm
423 If you want to use a single root file system that works cleanly for
424 domain0 and domains>0, one trick is to use different 'init' run
425 levels. For example, on the Xen Demo CD we use run level 3 for domain
426 0, and run level 4 for domains>0. This enables different startup
427 scripts to be run in depending on the run level number passed on the
428 kernel command line.
430 Xenolinux kernels can be built to use runtime loadable modules just
431 like normal linux kernels. Modules should be installed under
432 /lib/modules in the normal way.
434 If there's some kernel feature that hasn't been built into our default
435 kernel, there's a pretty good change that if its a non-hardware
436 related option you'll just be able to enable it and rebuild. If its
437 not on the xconfig menu, hack the arch/xeno/config.in to put the menu
438 back in.
440 If you're going to use the link local 169.254.1.x addresses to
441 communicate between VMs, there are a couple of other issues to watch
442 out for. RH9 appears to have a bug where by default it configures the
443 loopback interface with a 169.254 address, which stops it working
444 properly on eth0 for communicating with other domains.
446 This utterly daft RH9 behaviour can be stopped by appending
447 "NOZEROCONF=yes" to /etc/sysconfig/networking-scripts/ifcfg-lo
449 If you're going to use NFS root files systems mounted either from an
450 external server or from domain0 there are a couple of other gotchas.
451 The default /etc/sysconfig/iptables rules block NFS, so part way
452 through the boot sequence things will suddenly go dead.
454 If you're planning on having a separate NFS /usr partition, the RH9
455 boot scripts don't make life easy, as they attempt to mount NFS file
456 systems way to late in the boot process. The easiest way I found to do
457 this was to have a '/linuxrc' script run ahead of /sbin/init that
458 mounts /usr:
459 #!/bin/bash
460 /sbin/ipconfig lo
461 /sbin/portmap
462 /bin/mount /usr
463 exec /sbin/init "$@" <>/dev/console 2>&1
465 The one slight complication with the above is that /sbib/portmap is
466 dynamically linked against /usr/lib/libwrap.so.0 Since this is in
467 /usr, it won't work. I solved this by copying the file (and link)
468 below the /usr mount point, and just let the file be 'covered' when
469 the mount happens.
471 In some installations, where a shared read-only /usr is being used, it
472 may be desirable to move other large directories over into the
473 read-only /usr. For example, on the XenDemoCD we replace /bin /lib and
474 /sbin with links into /usr/root/bin /usr/root/lib and /usr/root/sbin
475 respectively. This creates other problems for running the /linuxrc
476 script, requiring bash, portmap, mount, ifconfig, and a handful of
477 other shared libraries to be copied below the mount point. I guess I
478 should have written a little statically linked C program...
482 Description of how the XenDemoCD boots
483 ======================================
485 1. Grub is used to load Xen, a XenoLinux kernel, and an initrd (initial
486 ram disk). [The source of the version of Grub used is in /usr/local/src]
488 2. the init=/linuxrc command line causes linux to execute /linuxrc in
489 the initrd.
491 3. the /linuxrc file attempts to mount the CD by trying the likely
492 locations : /dev/hd[abcd].
494 4. it then creates a 'tmpfs' file system and untars the
495 'XenDemoCD/root.tar.gz' file into the tmpfs. This contains hopefully
496 all the files that need to be mutable (this would be so much easier
497 if Linux supported 'stacked' or union file systems...)
499 5. Next, /linuxrc uses the pivot_root call to change the root file
500 system to the tmpfs, with the CD mounted as /usr.
502 6. It then invokes /sbin/init in the tmpfs and the boot proceeds
503 normally.
506 Building your own version of the XenDemoCD
507 ==========================================
509 The 'live ISO' version of RedHat is based heavily on Peter Anvin's
510 SuperRescue CD version 2.1.2 and J. McDaniel's Plan-B:
512 http://www.kernel.org/pub/dist/superrescue/v2/
513 http://projectplanb.org/
515 Since Xen uses a "multiboot" image format, it was necessary to change
516 the bootloader from isolinux to Grub0.93 with Leonid Lisovskiy's
517 <lly@pisem.net> grub.0.93-iso9660.patch
519 The Xen Demo CD contains all of the build scripts that were used to
520 create it, so it is possible to 'unpack' the current iso, modifiy it,
521 then build a new iso. The procedure for doing so is as follows:
523 First, mount either the CD, or the iso image of the CD:
525 mount /dev/cdrom /mnt/cdrom
526 or:
527 mount -o loop xendemo-1.0.iso /mnt/cdrom
529 cd to the directory you want to 'unpack' the iso into then run the
530 unpack script:
532 cd /local/xendemocd
533 /mnt/cdrom/XenDemoCD/unpack-iso.sh
535 The result is a 'build' directory containing the file system tree
536 under the 'root' directory. e.g. /local/xendemocd/build/root
538 To add or remove rpms, its possible to use 'rpm' with the --root
539 option to set the path. For more complex changes, it easiest to boot a
540 machine using using the tree via NFS root. Before doing this, you'll
541 need to edit fstab to comment out the seperate mount of /usr.
543 One thing to watch out for: as part of the CD build process, the
544 contents of the 'rootpatch' tree gets copied over the existing 'root'
545 tree replacing various files. The intention of the rootpatch tree is
546 to contain the files that have been modified from the original RH
547 distribution (e.g. various /etc files). This was done to make it
548 easier to upgrade to newer RH versions in the future. The downside of
549 this is that if you edit an existing file in the root tree you should
550 check that you don't also need to propagate the change to the
551 rootpatch tree to avoid it being overwritten.
553 Once you've made the changes and want to build a new iso, here's the
554 procedure:
556 cd /local/xendemocd/build
557 echo '<put_your_name_here>' > Builder
558 ./make.sh put_your_version_id_here >../buildlog 2>&1
560 This process can take 30 mins even on a fast machine, but you should
561 eventually end up with an iso image in the build directory.
563 Notes:
565 root - the root of the file system heirarchy as presented to the
566 running system
568 rootpatch - contains files that have been modified from the standard
569 RH, and copied over the root tree as part of the build
570 procedure.
572 irtree - the file system tree that will go into the initrd (initial
573 ram disk)
575 work - a working directory used in the build process
577 usr - this should really be in 'work' as its created as part of the
578 build process. It contains the 'immutable' files that will
579 be served from the CD rather than the tmpfs containing the
580 contents of root.tar.gz. Some files that are normally in /etc
581 or /var that are large and actually unlikely to need changing
582 have been moved into /usr/root and replaced with links.
585 Ian Pratt
586 9 Sep 2003