ia64/xen-unstable

changeset 2370:71fed8fd3ef9

bitkeeper revision 1.1159.53.12 (412f4c61U-yc4lDEgsveNajSuY4F-A)

Merge labyrinth.cl.cam.ac.uk:/auto/groups/xeno/BK/xeno.bk
into labyrinth.cl.cam.ac.uk:/auto/groups/xeno/users/akw27/xeno-clone/xeno.bk
author akw27@labyrinth.cl.cam.ac.uk
date Fri Aug 27 14:59:45 2004 +0000 (2004-08-27)
parents ac03da8a0f29 0ee3f9767bb4
children 41dcf2060c59
files .rootkeys docs/blkif-drivers-explained.txt
line diff
     1.1 --- a/.rootkeys	Fri Aug 27 13:41:23 2004 +0000
     1.2 +++ b/.rootkeys	Fri Aug 27 14:59:45 2004 +0000
     1.3 @@ -12,6 +12,7 @@ 40083bb4LVQzRqA3ABz0__pPhGNwtA docs/HOWT
     1.4  4021053fmeFrEyPHcT8JFiDpLNgtHQ docs/HOWTOs/Xen-HOWTO
     1.5  4022a73cgxX1ryj1HgS-IwwB6NUi2A docs/HOWTOs/XenDebugger-HOWTO
     1.6  3f9e7d53iC47UnlfORp9iC1vai6kWw docs/Makefile
     1.7 +412f4bd9sm5mCQ8BkrgKcAKZGadq7Q docs/blkif-drivers-explained.txt
     1.8  3f9e7d60PWZJeVh5xdnk0nLUdxlqEA docs/eps/xenlogo.eps
     1.9  3f9e7d63lTwQbp2fnx7yY93epWS-eQ docs/figs/dummy
    1.10  3f9e7d564bWFB-Czjv1qdmE6o0GqNg docs/interface.tex
     2.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     2.2 +++ b/docs/blkif-drivers-explained.txt	Fri Aug 27 14:59:45 2004 +0000
     2.3 @@ -0,0 +1,477 @@
     2.4 +=== How the Blkif Drivers Work ===
     2.5 +Andrew Warfield
     2.6 +andrew.warfield@cl.cam.ac.uk
     2.7 +
     2.8 +The intent of this is to explain at a fairly detailed level how the
     2.9 +split device drivers work in Xen 1.3 (aka 2.0beta).  The intended
    2.10 +audience for this, I suppose, is anyone who intends to work with the
    2.11 +existing blkif interfaces and wants something to help them get up to
    2.12 +speed with the code in a hurry.  Secondly though, I hope to break out
    2.13 +the general mechanisms that are used in the drivers that are likely to
    2.14 +be necessary to implement other drivers interfaces.
    2.15 +
    2.16 +As a point of warning before starting, it is worth mentioning that I
    2.17 +anticipate much of the specifics described here changing in the near
    2.18 +future.  There has been talk about making the blkif protocol
    2.19 +a bit more efficient than it currently is.  Keir's addition of grant
    2.20 +tables will change the current remapping code that is used when shared
    2.21 +pages are initially set up.
    2.22 +
    2.23 +Also, writing other control interface types will likely need support
    2.24 +from Xend, which at the moment has a steep learning curve... this
    2.25 +should be addressed in the future.
    2.26 +
    2.27 +For more information on the driver model as a whole, read the
    2.28 +"Reconstructing I/O" technical report
    2.29 +(http://www.cl.cam.ac.uk/Research/SRG/netos/papers/2004-xenngio.pdf).
    2.30 +
    2.31 +==== High-level structure of a split-driver interface ====
    2.32 +
    2.33 +Why would you want to write a split driver in the first place?  As Xen
    2.34 +is a virtual machine manager and focuses on isolation as an initial
    2.35 +design principle, it is generally considered unwise to share physical
    2.36 +access to devices across domains.  The reasons for this are obvious:
    2.37 +when device resources are shared, misbehaving code or hardware can
    2.38 +result in the failure of all of the client applications.  Moreover, as
    2.39 +virtual machines in Xen are entire OSs, standard device drives that
    2.40 +they might use cannot have multiple instantiations for a single piece
    2.41 +of hardware.  In light of all this, the general approach in Xen is to
    2.42 +give a single virtual machine hardware access to a device, and where
    2.43 +other VMs want to share the device, export a higher-level interface to
    2.44 +facilitate that sharing.  If you don't want to share, that's fine.
    2.45 +There are currently Xen users actively exploring running two
    2.46 +completely isolated X-Servers on a Xen host, each with it's own video
    2.47 +card, keyboard, and mouse.  In these situations, the guests need only
    2.48 +be given physical access to the necessary devices and left to go on
    2.49 +their own.  However, for devices such as disks and network interfaces,
    2.50 +where sharing is required, the split driver approach is a good
    2.51 +solution.
    2.52 +
    2.53 +The structure is like this:
    2.54 +
    2.55 +   +--------------------------+  +--------------------------+
    2.56 +   | Domain 0 (privileged)    |  | Domain 1 (unprivileged) |
    2.57 +   |                          |  |                          |
    2.58 +   | Xend ( Application )     |  |                          |
    2.59 +   | Blkif Backend Driver     |  | Blkif Frontend Driver    |
    2.60 +   | Physical Device Driver   |  |                          |
    2.61 +   +--------------------------+  +--------------------------+
    2.62 +   +--------------------------------------------------------+
    2.63 +   |                X       E       N                       |
    2.64 +   +--------------------------------------------------------+
    2.65 +
    2.66 +
    2.67 +The Blkif driver is in two parts, which we refer to as frontend (FE)
    2.68 +and a backend (BE).  Together, they serve to proxy device requests
    2.69 +between the guest operating system in an unprivileged domain, and the
    2.70 +physical device driver in the physical domain.  An additional benefit
    2.71 +to this approach is that the FE driver can provide a single interface
    2.72 +for a whole class of physical devices.  The blkif interface mounts
    2.73 +IDE, SCSI, and our own VBD-structured disks, independent of the
    2.74 +physical driver underneath.  Moreover, supporting additional OSs only
    2.75 +requires that a new FE driver be written to connect to the existing
    2.76 +backend.
    2.77 +
    2.78 +==== Inter-Domain Communication Mechanisms ====
    2.79 +
    2.80 +===== Event Channels =====
    2.81 +
    2.82 +Before getting into the specifics of the block interface driver, it is
    2.83 +worth discussing the mechanisms that are used to communicate between
    2.84 +domains.  Two mechanisms are used to allow the construction of
    2.85 +high-performance drivers: event channels and shared-memory rings.
    2.86 +
    2.87 +Event channels are an asynchronous interdomain notification
    2.88 +mechanism.  Xen allows channels to be instantiated between two
    2.89 +domains, and domains can request that a virtual irq be attached to
    2.90 +notifications on a given channel.  The result of this is that the
    2.91 +frontend domain can send a notification on an event channel, resulting
    2.92 +in an interrupt entry into the backend at a later time.
    2.93 +
    2.94 +The event channel between two domains is instantiated in the Xend code
    2.95 +during driver startup (described later).  Xend's channel.py
    2.96 +(tools/python/xen/xend/server/channel.py) defines the function
    2.97 +
    2.98 +
    2.99 +def eventChannel(dom1, dom2):
   2.100 +    return xc.evtchn_bind_interdomain(dom1=dom1, dom2=dom2)
   2.101 +
   2.102 +
   2.103 +which maps to xc_evtchn_bind_interdomain() in tools/libxc/xc_evtchn.c,
   2.104 +which in turn generates a hypercall to Xen to patch the event channel
   2.105 +between the domains.  Only a privileged domain can request the
   2.106 +creation of an event channel.
   2.107 +
   2.108 +Once the event channel is created in Xend, its ends are passed to both the
   2.109 +front and backend domains over the control channel.  The end that is
   2.110 +passed to a domain is just an integer "port" uniquely identifying the
   2.111 +event channel's local connection to that domain.  An example of this
   2.112 +setup code is in linux-2.6.x/drivers/xen/blkfront/blkfront.c in
   2.113 +blkif_status_change, which receives several status change events as
   2.114 +the driver starts up.  It is passed an event channel end in a
   2.115 +BLKIF_INTERFACE_STATUS_CONNECTED message, and patches it in like this:
   2.116 +
   2.117 +
   2.118 +   blkif_evtchn = status->evtchn;
   2.119 +   blkif_irq    = bind_evtchn_to_irq(blkif_evtchn);
   2.120 +   if ( (rc = request_irq(blkif_irq, blkif_int, 
   2.121 +                          SA_SAMPLE_RANDOM, "blkif", NULL)) )
   2.122 +       printk(KERN_ALERT"blkfront request_irq failed (%ld)\n",rc);
   2.123 +
   2.124 +
   2.125 +This code associates a virtual irq with the event channel, and
   2.126 +attaches the function blkif_int() as an interrupt handler for that
   2.127 +irq.  blkif_int() simply handles the notification and returns, it does
   2.128 +not need to interact with the channel at all.
   2.129 +
   2.130 +An example of generating a notification can also be seen in blkfront.c:
   2.131 +
   2.132 +
   2.133 +static inline void flush_requests(void)
   2.134 +{
   2.135 +    DISABLE_SCATTERGATHER();
   2.136 +    wmb(); /* Ensure that the frontend can see the requests. */
   2.137 +    blk_ring->req_prod = req_prod;
   2.138 +    notify_via_evtchn(blkif_evtchn);
   2.139 +}
   2.140 +}}}
   2.141 +
   2.142 +notify_via_evtchn issues a hypercall to set the event waiting flag on
   2.143 +the other domain's end of the channel.
   2.144 +
   2.145 +===== Communication Rings =====
   2.146 +
   2.147 +Event channels are strictly a notification mechanism between domains.
   2.148 +To move large chunks of data back and forth, Xen allows domains to
   2.149 +share pages of memory.  We use communication rings as a means of
   2.150 +managing access to a shared memory page for message passing between
   2.151 +domains.  These rings are not explicitly a mechanism of Xen, which is
   2.152 +only concerned with the actual sharing of the page and not how it is
   2.153 +used, they are however worth discussing as they are used in many
   2.154 +places in the current code and are a useful model for communicating
   2.155 +across a shared page.
   2.156 +
   2.157 +A shared page is set up by a guest first allocating and passing the
   2.158 +address of a page in its own address space to the backend driver.  
   2.159 +
   2.160 +
   2.161 +   blk_ring = (blkif_ring_t *)__get_free_page(GFP_KERNEL);
   2.162 +   blk_ring->req_prod = blk_ring->resp_prod = resp_cons = req_prod = 0;
   2.163 +   ...
   2.164 +   /* Construct an interface-CONNECT message for the domain controller. */
   2.165 +   cmsg.type      = CMSG_BLKIF_FE;
   2.166 +   cmsg.subtype   = CMSG_BLKIF_FE_INTERFACE_CONNECT;
   2.167 +   cmsg.length    = sizeof(blkif_fe_interface_connect_t);
   2.168 +   up.handle      = 0;
   2.169 +   up.shmem_frame = virt_to_machine(blk_ring) >> PAGE_SHIFT;
   2.170 +   memcpy(cmsg.msg, &up, sizeof(up));  
   2.171 +
   2.172 +
   2.173 +blk_ring will be the shared page.  The producer and consumer pointers
   2.174 +are then initialised (these will be discussed soon), and then the
   2.175 +machine address of the page is send to the backend via a control
   2.176 +channel to Xend.  This control channel itself uses the notification
   2.177 +and shared memory mechanisms described here, but is set up for each
   2.178 +domain automatically at startup.
   2.179 +
   2.180 +The backend, which is a privileged domain then takes the page address
   2.181 +and maps it into its own address space (in
   2.182 +linux26/drivers/xen/blkback/interface.c:blkif_connect()):
   2.183 +
   2.184 +
   2.185 +void blkif_connect(blkif_be_connect_t *connect)
   2.186 +
   2.187 +   ...
   2.188 +   unsigned long shmem_frame = connect->shmem_frame;
   2.189 +   ...
   2.190 +
   2.191 +   if ( (vma = get_vm_area(PAGE_SIZE, VM_IOREMAP)) == NULL )
   2.192 +   {
   2.193 +      connect->status = BLKIF_BE_STATUS_OUT_OF_MEMORY;
   2.194 +      return;
   2.195 +   }
   2.196 +
   2.197 +   prot = __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED);
   2.198 +   error = direct_remap_area_pages(&init_mm, VMALLOC_VMADDR(vma->addr),
   2.199 +                                   shmem_frame<<PAGE_SHIFT, PAGE_SIZE,
   2.200 +                                   prot, domid);
   2.201 +
   2.202 +   ...
   2.203 +
   2.204 +   blkif->blk_ring_base = (blkif_ring_t *)vma->addr
   2.205 +}}}
   2.206 +
   2.207 +The machine address of the page is passed in the shmem_frame field of
   2.208 +the connect message.  This is then mapped into the virtual address
   2.209 +space of the backend domain, and saved in the blkif structure
   2.210 +representing this particular backend connection.
   2.211 +
   2.212 +NOTE:  New mechanisms will be added very shortly to allow domains to
   2.213 +explicitly grant access to their pages to other domains.  This "grant
   2.214 +table" support is in the process of being added to the tree, and will
   2.215 +change the way a shared page is set up.  In particular, it will remove
   2.216 +the need of the remapping domain to be privileged.
   2.217 +
   2.218 +Sending data across shared rings:
   2.219 +
   2.220 +Shared rings avoid the potential for write interference between
   2.221 +domains in a very cunning way.  A ring is partitioned into a request
   2.222 +and a response region, and domains only work within their own space.
   2.223 +This can be thought of as a double producer-consumer ring -- the ring
   2.224 +is described by four pointers into a circular buffer of fixed-size
   2.225 +records.  Pointers may only advance, and may not pass one another.
   2.226 +
   2.227 +
   2.228 +                          rsp_cons----+
   2.229 +                                      V
   2.230 +           +----+----+----+----+----+----+----+
   2.231 +           |    |    |    free      |RSP1|RSP2|
   2.232 +           +----+----+----+----+----+----+----+
   2.233 + req_prod->|    |       -------->        |RSP3|
   2.234 +           +----+                        +----+
   2.235 +           |REQ8|                        |    |<-rsp_prod
   2.236 +           +----+                        +----+
   2.237 +           |REQ7|                        |    |
   2.238 +           +----+                        +----+
   2.239 +           |REQ6|       <--------        |    |
   2.240 +           +----+----+----+----+----+----+----+
   2.241 +           |REQ5|REQ4|    free      |    |    |
   2.242 +           +----+----+----+----+----+----+----+
   2.243 +  req_cons---------^
   2.244 +
   2.245 +
   2.246 +
   2.247 +By adopting the convention that every request will receive a response,
   2.248 +not all four pointers need be shared and flow control on the ring
   2.249 +becomes very easy to manage.  Each domain manages its own
   2.250 +consumer pointer, and the two producer pointers are visible to both (Xen/include/hypervisor-ifs/io/blkif.h):
   2.251 +
   2.252 +
   2.253 +
   2.254 +/* NB. Ring size must be small enough for sizeof(blkif_ring_t) <=PAGE_SIZE.*/
   2.255 +  #define BLKIF_RING_SIZE        64
   2.256 +
   2.257 +  ...
   2.258 +
   2.259 +/*
   2.260 + * We use a special capitalised type name because it is _essential_ that all
   2.261 + * arithmetic on indexes is done on an integer type of the correct size.
   2.262 + */
   2.263 +typedef u32 BLKIF_RING_IDX;
   2.264 +
   2.265 +/*
   2.266 + * Ring indexes are 'free running'. That is, they are not stored modulo the
   2.267 + * size of the ring buffer. The following macro converts a free-running counter
   2.268 + * into a value that can directly index a ring-buffer array.
   2.269 + */
   2.270 +#define MASK_BLKIF_IDX(_i) ((_i)&(BLKIF_RING_SIZE-1))
   2.271 +
   2.272 +typedef struct {
   2.273 +    BLKIF_RING_IDX req_prod;  /*  0: Request producer. Updated by front-end. */
   2.274 +    BLKIF_RING_IDX resp_prod; /*  4: Response producer. Updated by back-end. */
   2.275 +    union {                   /*  8 */
   2.276 +        blkif_request_t  req;
   2.277 +        blkif_response_t resp;
   2.278 +    } PACKED ring[BLKIF_RING_SIZE];
   2.279 +} PACKED blkif_ring_t;
   2.280 +
   2.281 +
   2.282 +
   2.283 +As shown in the diagram above, the rules for using a shared memory
   2.284 +ring are simple.  
   2.285 +
   2.286 + 1. A ring is full when a domain's producer and consumer pointers are
   2.287 +    equal (e.g. req_prod == resp_cons).  In this situation, the
   2.288 +    consumer pointer must be advanced.  Furthermore, if the consumer
   2.289 +    pointer is equal to the other domain's producer pointer,
   2.290 +    (e.g. resp_cons = resp_prod), then the other domain has all the
   2.291 +    buffers.
   2.292 +
   2.293 +2. Producer pointers point to the next buffer that will be written to.
   2.294 +   (So blk_ring[MASK_BLKIF_IDX(req_prod)] should not be consumed.)
   2.295 +
   2.296 +3. Consumer pointers point to a valid message, so long as they are not
   2.297 +   equal to the associated producer pointer.
   2.298 +
   2.299 +4. A domain should only ever write to the message pointed
   2.300 +   to by its producer index, and read from the message at it's
   2.301 +   consumer.  More generally, the domain may be thought of to have
   2.302 +   exclusive access to the messages between its consumer and producer,
   2.303 +   and should absolutely not read or write outside this region.
   2.304 +
   2.305 +In general, drivers keep a private copy of their producer pointer and
   2.306 +then set the shared version when they are ready for the other end to
   2.307 +process a set of messages.  Additionally, it is worth paying attention
   2.308 +to the use of memory barriers (rmb/wmb) in the code, to ensure that
   2.309 +rings that are shared across processors behave as expected.
   2.310 +
   2.311 +==== Structure of the Blkif Drivers ====
   2.312 +
   2.313 +Now that the communications primitives have been discussed, I'll
   2.314 +quickly cover the general structure of the blkif driver.  This is
   2.315 +intended to give a high-level idea of what is going on, in an effort
   2.316 +to make reading the code a more approachable task.
   2.317 +
   2.318 +There are three key software components that are involved in the blkif
   2.319 +drivers (not counting Xen itself).  The frontend and backend driver,
   2.320 +and Xend, which coordinates their initial connection.  Xend may also
   2.321 +be involved in control-channel signalling in some cases after startup,
   2.322 +for instance to manage reconnection if the backend is restarted.
   2.323 +
   2.324 +===== Frontend Driver Structure =====
   2.325 +
   2.326 +The frontend domain uses a single event channel and a shared memory
   2.327 +ring to trade control messages with the backend.  These are both setup
   2.328 +during domain startup, which will be discussed shortly.  The shared
   2.329 +memory ring is called blkif_ring, and the private ring indexes are
   2.330 +resp_cons, and req_prod.  The ring is protected by blkif_io_lock.
   2.331 +Additionally, the frontend keeps a list of outstanding requests in
   2.332 +rec_ring[].  These are uniquely identified by a guest-local id number,
   2.333 +which is associated with each request sent to the backend, and
   2.334 +returned with the matching responses.  Information about the actual
   2.335 +disks are stored in major_info[], of which only the first nr_vbds
   2.336 +entries are valid.  Finally, the global 'recovery' indicates that the
   2.337 +connection between the backend and frontend drivers has been broken
   2.338 +(possibly due to a backend driver crash) and that the frontend is in
   2.339 +recovery mode, in which case it will attempt to reconnect and reissue
   2.340 +outstanding requests.
   2.341 +
   2.342 +The frontend driver is single-threaded and after setup is entered only
   2.343 +through three points:  (1) read/write requests from the XenLinux guest
   2.344 +that it is a part of, (2) interrupts from the backend driver on its
   2.345 +event channel (blkif_int()), and (3) control messages from Xend
   2.346 +(blkif_ctrlif_rx).
   2.347 +
   2.348 +===== Backend Driver Structure =====
   2.349 +
   2.350 +The backend driver is slightly more complex as it must manage any
   2.351 +number of concurrent frontend connections.  For each domain it
   2.352 +manages, the backend driver maintains a blkif structure, which
   2.353 +describes all the connection and disk information associated with that
   2.354 +particular domain.  This structure is associated with the interrupt
   2.355 +registration, and allows the backend driver to have immediate context
   2.356 +when it takes a notification from some domain.
   2.357 +
   2.358 +All of the blkif structures are stored in a hash table (blkif_hash),
   2.359 +which is indexed by a hash of the domain id, and a "handle", really a
   2.360 +per-domain blkif identifier, in case it wants to have multiple connections.
   2.361 +
   2.362 +The per-connection blkif structure is of type blkif_t.  It contains
   2.363 +all of the communication details (event channel, irq, shared memory
   2.364 +ring and indexes), and blk_ring_lock, which is the backend mutex on
   2.365 +the shared ring.  The structure also contains vbd_rb, which is a
   2.366 +red-black tree, containing an entry for each device/partition that is
   2.367 +assigned to that domain.  This structure is filled by xend passing
   2.368 +disk information to the backend at startup, and is protected by
   2.369 +vbd_lock.  Finally, the blkif struct contains a status field, which
   2.370 +describes the state of the connection.
   2.371 +
   2.372 +The backend driver spawns a kernel thread at startup
   2.373 +(blkio_schedule()), which handles requests to and from the actual disk
   2.374 +device drivers.  This scheduler thread maintains a list of blkif
   2.375 +structures that have pending requests, and services them round-robin
   2.376 +with a maximum per-round request limit.  blkifs are added to the list
   2.377 +in the interrupt handler (blkif_be_int()) using
   2.378 +add_to_blkdev_list_tail(), and removed in the scheduler loop after
   2.379 +calling do_block_io_op(), which processes a batch of requests.  The
   2.380 +scheduler thread is explicitly activated at several points in the code
   2.381 +using maybe_trigger_blkio_schedule().
   2.382 +
   2.383 +Pending requests between the backend driver and the physical device
   2.384 +drivers use another ring, pending_ring.  Requests are placed in this
   2.385 +ring in the scheduler thread and issued to the device.  A completion
   2.386 +callback, end_block_io_op, indicates that requests have been serviced
   2.387 +and generates a response on the appropriate blkif ring.  pending
   2.388 +reqs[] stores a list of outstanding requests with the physical drivers.
   2.389 +
   2.390 +So, control entries to the backend are (1) the blkio scheduler thread,
   2.391 +which sends requests to the real device drivers, (2) end_block_io_op,
   2.392 +which is called as serviced requests complete, (3) blkif_be_int()
   2.393 +handles notifications from the frontend drivers in other domains, and
   2.394 +(4) blkif_ctrlif_rx() handles control messages from xend.
   2.395 +
   2.396 +==== Driver Startup ====
   2.397 +
   2.398 +Prior to starting a new guest using the frontend driver, the backend
   2.399 +will have been started in a privileged domain.  The backend
   2.400 +initialisation code initialises all of its data structures, such as
   2.401 +the blkif hash table, and starts the scheduler thread as a kernel
   2.402 +thread. It then sends a driver status up message to let xend know it
   2.403 +is ready to take frontend connections.
   2.404 +
   2.405 +When a new domain that uses the blkif frontend driver is started,
   2.406 +there are a series of interactions between it, xend, and the specified
   2.407 +backend driver.  These interactions are as follows:
   2.408 +
   2.409 +The domain configuration given to xend will specify the backend domain
   2.410 +and disks that the new guest is to use.  Prior to actually running the
   2.411 +domain, xend and the backend driver interact to setup the initial
   2.412 +blkif record in the backend.
   2.413 +
   2.414 +(1) Xend sends a BLKIF_BE_CREATE message to backend.
   2.415 +
   2.416 +  Backend does blkif_create(), having been passed FE domid and handle.
   2.417 +  It creates and initialises a new blkif struct, and puts it in the
   2.418 +  hash table.
   2.419 +  It then returns a STATUS_OK response to xend.
   2.420 +
   2.421 +(2) Xend sends a BLKIF_BE_VBD_CREATE message to the backend.
   2.422 + 
   2.423 +  Backend adds a vbd entry in the red-black tree for the
   2.424 +  specified (dom, handle) blkif entry.
   2.425 +  Sends a STATUS_OK response.
   2.426 +
   2.427 +(3) Xend sends a BLKIF_BE_VBD_GROW message to the backend.
   2.428 +
   2.429 +  Backend takes the physical device information passed in the 
   2.430 +  message and assigns them to the newly created vbd struct.
   2.431 +
   2.432 +(2) and (3) repeat as any additional devices are added to the domain.
   2.433 +
   2.434 +At this point, the backend has enough state to allow the frontend
   2.435 +domain to start.  The domain is run, and eventually gets to the
   2.436 +frontend driver initialisation code.  After setting up the frontend
   2.437 +data structures, this code continues the communications with xend and
   2.438 +the backend to negotiate a connection:
   2.439 +
   2.440 +(4) Frontend sends Xend a BLKIF_FE_DRIVER_STATUS_CHANGED message.
   2.441 +
   2.442 +  This message tells xend that the driver is up.  The init function
   2.443 +  now spin-waits until driver setup is complete in order to prevent
   2.444 +  Linux from attempting to boot before the disks are connected.
   2.445 +
   2.446 +(5) Xend sends the frontend an INTERFACE_STATUS_CHANGED message
   2.447 +
   2.448 +  This message specifies that the interface is now disconnected
   2.449 +  (instead of closed).
   2.450 +  The domain updates it's state, and allocates the shared blk_ring
   2.451 +  page.  Next, 
   2.452 +
   2.453 +(6) Frontend sends Xend a BLKIF_INTERFACE_CONNECT message
   2.454 +
   2.455 +  This message specifies the domain and handle, and includes the
   2.456 +  address of the newly created page.
   2.457 +
   2.458 +(7) Xend sends the backend a BLKIF_BE_CONNECT message
   2.459 +
   2.460 +  The backend fills in the blkif connection information, maps the
   2.461 +  shared page, and binds an irq to the event channel.
   2.462 +  
   2.463 +(8) Xend sends the frontend an INTERFACE_STATUS_CHANGED message
   2.464 +
   2.465 +  This message takes the frontend driver to a CONNECTED state, at
   2.466 +  which point it binds an irq to the event channel and calls
   2.467 +  xlvbd_init to initialise the individual block devices.
   2.468 +
   2.469 +The frontend Linux is stall spin waiting at this point, until all of
   2.470 +the disks have been probed.  Messaging now is directly between the
   2.471 +front and backend domain using the new shared ring and event channel.
   2.472 +
   2.473 +(9) The frontend sends a BLKIF_OP_PROBE directly to the backend.
   2.474 +
   2.475 +  This message includes a reference to an additional page, that the
   2.476 +  backend can use for it's reply.  The backend responds with an array
   2.477 +  of the domains disks (as vdisk_t structs) on the provided page.
   2.478 +
   2.479 +The frontend now initialises each disk, calling xlvbd_init_device()
   2.480 +for each one.