view docs/xend.tex @ 2570:30a9b33481dc

bitkeeper revision 1.1159.90.2 (415be9d5hTw1zLV9fA-AYcekmwhMwg)

Discard devices early for local migrate.
author mjw@wray-m-3.hpl.hp.com
date Thu Sep 30 11:11:17 2004 +0000 (2004-09-30)
parents 0b883cd8b325
children 6ef02dbe3f30
line source
1 % -*- mode: LaTeX -*-
2 \def\seca{\chapter}
3 \def\secb{\section}
4 \def\secc{\subsection}
5 \def\secd{\subsubsection}
6 \def\refa{chapter}
7 \def\refb{section}
8 \def\refc{section}
9 \def\refd{section}
11 %\def\seca{\section}
12 %\def\secb{\subsection}
13 %\def\secc{\subsubsection}
14 %\def\refa{section}
15 %\def\refb{section}
16 %\def\refc{section}
18 \documentclass[11pt,twoside,final,openright]{xenstyle}
19 \usepackage{a4,graphicx,setspace}
20 \setstretch{1.15}
21 \input{style.tex}
23 \begin{document}
26 \pagestyle{empty}
27 \begin{center}
28 \vspace*{\fill}
29 \includegraphics{eps/xenlogo.eps}
30 \vfill
31 \vfill
32 \vfill
33 \begin{tabular}{l}
34 {\Huge \bf Xend} \\[4mm]
35 {\huge Xen v2.0 for x86} \\[80mm]
37 {\Large Xen is Copyright (c) 2004, The Xen Team} \\[3mm]
38 {\Large University of Cambridge, UK} \\[20mm]
39 {\large Last updated 30 August 2004}
40 \end{tabular}
41 \vfill
42 \end{center}
43 \cleardoublepage
46 \pagestyle{plain}
47 \pagenumbering{roman}
48 { \parskip 0pt plus 1pt
49 \tableofcontents }
50 \cleardoublepage
52 \pagenumbering{arabic}
53 \raggedbottom
54 \widowpenalty=10000
55 \clubpenalty=10000
56 \parindent=0pt
57 \renewcommand{\topfraction}{.8}
58 \renewcommand{\bottomfraction}{.8}
59 \renewcommand{\textfraction}{.2}
60 \renewcommand{\floatpagefraction}{.8}
62 \setstretch{1.15}
64 \seca{Introduction}
65 Xend is the control daemon used to manage a machine running the Xen hypervisor.
66 Xend is responsible for creating and destroying domains and managing their
67 resources, such as virtual block devices and virtual network interfaces.
69 Xend exists because the Xen hypervisor itself only manages the memory image
70 of a domain and its scheduling. Xen provides the event channels that connect
71 a domain to its devices, but is intentionally not involved in setting them up.
73 Xend runs as a daemon in the privileged domain 0 and uses a low-level api
74 to communicate with Xen via the domain 0 kernel. Xend exports its control
75 interface to its clients using HTTP. Most programming languages have
76 HTTP client libraries, so this interface can be used from most popular
77 languages, for example Python, Perl, C, Java.
78 Xend itself is written in Python, as are most of the Xen tools.
80 The xend interface is intended to be a complete interface for the creation
81 and management of domains. It supports domain creation, shutdown, reboot,
82 destruction, save, restore and migration.
84 When xend creates a domain it creates the domain memory image and communicates
85 with the device driver domain(s) to configure the devices for the domain.
86 This sets up connections between the domain and backend device controllers
87 in the driver domain. When a domain shuts down its memory image cannot be fully released
88 unless its backend devices are released and disconnected. This is done by xend.
89 In order to protect against loss of this information when xend is restarted,
90 xend maintains a persistent database of domain configurations. This allows
91 xend to be stopped and restarted without loss of configuration information.
92 For example, in order to upgrade the xend software.
94 \seca{Domain lifecycle}
95 \secb{Domain creation}
96 Xend is instructed to create a domain by posting a domain\_create message to it,
97 containing the domain configuration to be instantiated. The domain configuration
98 is in sxp format and is as far as possible {\em fully-bound}, that is, all
99 parameters are fully-specified. The domain configuration is saved in the filesystem
100 so that it can be reused later if necessary.
102 The domain configuration specifies the domain name, memory size, kernel image
103 and parameters, and all the domain devices. Xend uses the Xen api to create
104 the domain memory image, and then {\em builds} the memory image for the domain
105 using the kernel image. At this point the domain exists, but it cannot be run because
106 it has no devices. Xend then communicates with the device driver domain to create
107 the configured devices. Once the devices are created it sets up event channels
108 for them between the driver domain and the new domain, and notifies the new domain
109 that its devices are connected. At this point the domain can be started.
111 Xend is also responsible for managing domain consoles. When a domain is created,
112 xend sets up a console event channel to the domain, and creates a TCP listening port
113 for the domain console. When a connection is accepted to the port, xend
114 connects input and output for the port to the domain console channel.
116 \secb{Domain destruction}
117 When a domain terminates, because it has been shutdown or it has crashed, the
118 domain resources must be released so that the domain memory image can be
119 finally removed from xen. Xend monitors the domains, and is also signaled by
120 xen (using a VIRQ) when a domain exits. Xend examines the domain states and
121 determines which domains have exited. It then communicates with the driver domain
122 to release the devices for exited domains. Xend also closes any open console
123 connections and removes the TCP listeners for exited domains.
124 Once all devices have been released it instructs xen to destroy the memory image.
126 \secb{Domain restart}
127 Domain restart is the xen equivalent of a machine reboot. When a domain
128 exits because it has been shutdown in reboot mode, its exit code is reboot.
129 When examining domains to find those that have exited and destroy them,
130 xend detects those that have exited for reboot and does not completely destroy
131 them. It disconnects all their devices, and detaches the console listener
132 from its channel to the domain, but does not close it. Instead it schedules
133 a call to rebuild the domain from its configuration. This proceeds almost
134 identically to creating the domain, except that the console listener is
135 reused and connected to the new domain. This allows existing console
136 connections to remain connected across a domain restart. The restarted
137 domain keeps the same name and domain id.
139 The determination of when to restart a domain is in fact slightly more
140 complex than described above. A domain is configured with a
141 {\em restart mode}. If the restart mode is {\em onreboot}, the default,
142 restart happens when the domain is shutdown normally and
143 exits with code reboot. If the restart mode is {\em never} the domain is
144 not restarted. If the restart mode is {\em always} the domain is always
145 restarted, regardless of how it exited.
147 In order to prevent continual domain crash causing restart loops, xend
148 has a {\em minimum restart time}. Xend remembers when a domain was last
149 restarted and will fail a restart that happens inside the minimum
150 restart time.
152 \seca{Devices}
153 \secb{Virtual network interfaces}
154 Each virtual network interface (vif) has 2 parts: the font-end device in its domain,
155 and the back-end device in the driver domain. Usually the driver domain is domain 0,
156 and there is a linux network device corresponding to the vif. The linux device for
157 interface N on domain D is called vifD.N. When a packet is sent on the vif in the
158 domain the packet is received from the linux device. The linux devices are connected
159 to network interfaces using ethernet bridging.
161 The default setup is a bridge xen-br0, with eth0 connected to it, and the routes
162 for eth0 directed at xen-br0. This is controlled by the xend network setup script,
163 default {\tt /etc/xen/network}, which is run when xend starts.
165 When the vifs for a domain are created, a vif control script, default {\tt /etc/xen/vif-bridge},
166 is run to connect the vif to its bridge. The default script connects the vif
167 to xen-br0 and optionally sets up iptables rules to prevent IP address spoofing.
168 The bridge a vif is connected to can be defined in its configuration, and this is useful
169 for setting up virtual networks using several bridges.
171 \secb{Virtual block devices}
172 Virtual block devices in a domain are interfaces onto back-end device drivers
173 that export physical devices to domains. In the default configuration the back-end
174 driver is in domain 0 and can export any linux block device to a domain. This includes
175 physical disk partitions, LVM volumes and loopback mounts of files. In fact anything
176 that linux can represent as a block device can be exported to a domain as virtual
177 block device.
179 \seca{Xend invocation}
180 Xend is started (by root) using the command
181 \begin{verbatim}
182 xend start
183 \end{verbatim}
184 Xend can be stopped using
185 \begin{verbatim}
186 xend stop
187 \end{verbatim}
188 Xend must be started before any domains (apart from domain 0) can be created.
189 If you try to use the {\tt xm} tool when xend is not running you will get a
190 'connection refused' message.
192 \secb{Xend configuration}
193 Xend reads its own configuration from {\tt /etc/xen/xend-config.sxp}, which is
194 a sequence of s-expressions. The configuration parameters are:
195 \begin{itemize}
197 \item xend-port: Port xend should use for the HTTP interface (default 8000).
199 \item xend-address: Address xend should listen on.
200 Specifying 'localhost' prevents remote connections.
201 Specifying the empty string '' allows all connections, and is the default.
203 \item network-script: The script used to start/stop networking for xend (default network).
205 \item vif-bridge: The default bridge that virtual interfaces should be connected to
206 (default xen-br0).
208 \item vif-script: The default script used to control virtual interfaces
209 (default vif-bridge).
211 \item vif-antispoof: Whether iptables should be set up to prevent IP spoofing for
212 virtual interfaces (default yes).
213 \end{itemize}
215 Configuration scripts ({\it e.g.} for network-script) are looked for in {\tt /etc/xen}
216 unless their name begins with '/'.
218 Xend sends its log output to {\tt /var/log/xend.log}. This is a rotating logfile,
219 and logs are moved onto {\tt xend.log.1} {\it etc.} as they get large. Old logs may
220 be deleted.
222 \secb{Xend database}
223 Xend needs to make some data persistent, and it uses files under {\tt /var/xen/xend-db}
224 for this. The persistent data is stored in files in SXP format. Domain information
225 is saved when domains are created. When xend starts it reads the file {\tt /var/xen/lastboot}
226 (if it exists) to determine the last time the system was rebooted. It compares this time
227 with the last reboot time in {\tt wtmp} to determine if the system has been rebooted
228 since xend last ran. If the system has been rebooted xend removes all its saved data
229 that is not persistent across reboots (for example domain data).
231 \seca{Xend HTTP Interface}
232 The xend interface uses HTTP 1.1 \cite{http} as its transport.
233 Simple PUT and GET calls can encode parameters using the standard url-encoding
234 for parameters: MIME type {\tt application/x-www-form-urlencoded}.
235 When file upload is required, the {\tt multipart/form-data} encoding is used.
236 See the HTML 4.1 specification for details \cite{html}.
238 Xend functions as a webserver and supports two interfaces: one
239 for web-browsers and one for programs.
240 The web-browser interface returns replies in HTML and includes forms
241 for interactive operations such as stopping domains and creating domains
242 from an uploaded configuration. The programmatic interface usually returns replies
243 in s-expression format. Both interfaces are accessed
244 in exactly the same way over HTTP - the only difference is the data returned.
246 The webserver distinguishes browsers from programs using the {\tt User-Agent}
247 and {\tt Accept} headers in the HTTP request. If there is no {\tt User-Agent} or no
248 {\tt Acccept} header, or {\tt Accept} includes the type {\tt application/sxp}, the
249 webserver assumes the client is a program and returns SXP. Otherwise
250 it assumes the client is a webserver and returns HTML.
251 In some cases the return value is essentially a string, so {\tt Content-Type}
252 {\tt text/plain} is returned.
254 The HTTP api supported is listed below. All paths in it are relative to the
255 server root, for example {\tt http://localhost:8000/xend}.
256 As defined in the HTTP specification, we use GET for side-effect free
257 operations that may safely be repeated, and POST for operations with
258 side-effects. For each request we list the HTTP method (GET or POST),
259 the url relative to the server root, the operation name and arguments (if any).
260 The operation name is passed as request parameter 'op', and the arguments
261 are passed by name. Operation name and parameters can be encoded using either
262 encoding described above. We also list the corresponding api function from the
263 Python client interface in {\tt xen.xend.XendClient}.
265 \begin{itemize}
266 \item {\tt GET /},\\
267 {\tt xend()}:\\
268 Get list of urls under xend root.
270 \item {\tt GET /node},\\
271 {\tt xend\_node()}:\\
272 Get node information.
274 \item {\tt POST /node shutdown()},\\
275 {\tt xend\_node\_shutdown()}:\\
276 Shutdown the node
278 \item {\tt POST /node reboot()},\\
279 {\tt xend\_node\_reboot()}:\\
280 Reboot the node
282 \item {\tt POST /node notify()}:\\
283 Set node notification url
285 \item {\tt GET /node/dmesg},\\
286 {\tt xend\_node\_dmesg()}:\\
287 Get xen boot message.
289 \item {\tt GET /node/log},\\
290 {\tt xend\_node\_log()}:\\
291 Get xend log.
293 \item {\tt GET /domain}\\
294 {\tt xend\_domains()}:\\
295 Get list of domains.
297 \item {\tt POST /domain create(config)},\\
298 {\tt xend\_domain\_create(config)}:\\
299 Create a domain.
301 \item {\tt POST /domain restore(file)},\\
302 {\tt xend\_domain\_restore(filename)}:\\
303 Restore a saved domain.
305 \item {\tt GET /domain/<dom>},\\
306 {\tt xend\_domain(dom)}:\\
307 Get domain information.
309 \item {\tt POST /domain/[dom] configure(config)},\\
310 {\tt xend\_domain\_configure(dom, conf)}:\\
311 Configure an existing domain (for internal use by restore and migrate).
313 \item {\tt POST /domain/[dom] unpause()},\\
314 {\tt xend\_domain\_unpause(dom)}:\\
315 Start domain running
317 \item {\tt POST /domain/[dom] pause()},\\
318 {\tt xend\_domain\_pause(dom)}:\\
319 Stop domain running.
321 \item {\tt POST /domain/[dom] shutdown(reason)},\\
322 {\tt xend\_domain\_shutdown(dom, reason)}:\\
323 Shutdown domain, reason can be reboot, poweroff, halt.
325 \item {\tt POST /domain/[dom] destroy(reason)},\\
326 {\tt xend\_domain\_destroy(dom, reason)}:\\
327 Destroy domain, reason can be reboot, halt.
329 \item {\tt POST /domain/[dom] save(file)},\\
330 {\tt xend\_domain\_save(dom, filename)}:\\
331 Save a domain to a file.
333 \item {\tt POST /domain/[dom] migrate(dst)},\\
334 {\tt xend\_domain\_migrate(dom, dst)}:\\
335 Migrate a domain.
337 \item {\tt POST /domain/[dom] pincpu(cpu)},\\
338 {\tt xend\_domain\_pincpu(self, id, cpu)}\\:
339 Pin a domain to a cpu.
341 \item {\tt POST /domain/[dom] bvt\_set(mcuadv, warp, warpl, warpu)},\\
342 {\tt xend\_domain\_cpu\_bvt\_set(dom, mcuadv, warp, warpl, warpu)}:\\
343 Set BVT scheduler parameters.
345 \item {\tt POST /domain/[dom] fbvt\_set(mcuadv, warp, warpl, warpu)},\\
346 {\tt xend\_domain\_cpu\_fbvt\_set(dom, mcuadv, warp, warpl, warpu)}:\\
347 Set FBVT scheduler parameters.
349 \item {\tt POST /domain/[dom] atropos\_set(period, slice, latency, xtratime)},\\
350 {\tt xend\_domain\_cpu\_atropos\_set(dom, period, slice, latency, xtratime)}:\\
351 Set atropos scheduler parameters.
353 \item {\tt POST /domain/[dom] maxmem\_set(memory)},\\
354 {\tt xend\_domain\_maxmem\_set(dom, memory)}:\\
355 Set domain maximum memory limit.
357 \item {\tt POST /domain/[dom] device\_create(config)}\\
358 {\tt xend\_domain\_device\_create(dom, config)}:\\
359 Add a device to a domain.
361 \item {\tt POST /domain/[dom] device\_destroy(type, index)},\\
362 {\tt xend\_domain\_device\_destroy(dom, type, index)}:\\
363 Delete a device from a domain
365 \item {\tt GET /domain/[dom] vifs()},\\
366 {\tt xend\_domain\_vifs(dom)}:\\
367 Get virtual network interfaces.
369 \item {\tt GET /domain/[dom] vif(vif)},\\
370 {\tt xend\_domain\_vif(dom, vif)}:\\
371 Get virtual network interface.
373 \item {\tt GET /domain/[dom] vbds()},\\
374 {\tt xend\_domain\_vbds(dom)}:\\
375 Get virtual block devices.
377 \item {\tt GET /domain/[dom] vbd(vbd)},\\
378 {\tt xend\_domain\_vbd(dom, vbd)}:\\
379 Get virtual block device.
381 \item {\tt GET /console},\\
382 {\tt xend\_consoles()}:\\
383 Get list of consoles.
385 \item {\tt GET /console/[id]}\\
386 {\tt xend\_console(id)}:\\
387 Get information about a console.
389 \item {\tt GET /console/[id] disconnect()}\\
390 {\tt xend\_console\_disconnect(self, id)}:\\
391 Disconnect any console TCP connection.
393 \item {\tt GET /vnet}\\
394 {\tt xend\_vnets()}:\\
395 Get list of vnets (virtual networks).
397 \item {\tt GET /vnet/[id]}\\
398 {\tt xend\_vnet(id)}:\\
399 Get information about a virtual network.
401 \item {\tt POST /vnet create(config)}\\
402 {\tt xend\_vnet\_create(conf)}:\\
403 Create a vnet.
405 \item {\tt POST /vnet/[id] delete()}\\
406 {\tt xend\_vnet\_delete(id)}:\\
407 Delete a vnet.
409 \item {\tt POST /event inject(event)}\\
410 {\tt xend\_event\_inject(sxpr)}:\\
411 Inject an event.
413 \end{itemize}
415 \secb{Xend debugging interface}
416 Xend also listens on port 8001. Connecting to this port (for example via telnet)
417 allows access to some debugging functions:
418 \begin{itemize}
419 \item help: list functions
420 \item traceon: turn xend tracing on
421 \item traceoff: turn xend tracing off
422 \item quit: disconnect.
423 \item info: list console listeners, block and network device controllers.
424 \end{itemize}
426 When tracing is on xend logs all functions calls and exceptions to
427 {\tt /var/log/xend.trace}.
429 \begin{thebibliography}{99}
431 \bibitem{html}
432 HTML 4.01 Specification,\\
433 http://www.w3.org/TR/html4/,\\
434 W3C Recommendation, 24 December 1999.
436 \bibitem{http}
437 Hypertext Transfer Protocol -- HTTP/1.1,\\
438 http://www.ietf.org/rfc/rfc2616.txt,\\
439 RFC 2616, IETF 1999.
441 \bibitem{ssh}
442 http://www.openssh.org.
444 \bibitem{stunnel}
445 http://www.stunnel.org.
447 \end{thebibliography}
448 \end{document}