Ben Pfaff [Thu, 8 Jul 2010 13:25:51 +0000 (14:25 +0100)]
[PATCH] xenserver: Set xs-network-uuids for both datapath and VLAN bridges.
From c0a50086d9e60995b17bdd8a4e1f150caba337d5 Mon Sep 17 00:00:00 2001
Date: Mon, 22 Feb 2010 16:41:19 -0800
It makes sense to set xs-network-uuids for every bridge, so this commit
does so. It also makes the code a bit more readable by factoring logic
out into a new function.
Signed-off-by: Ben Pfaff <blp@nicira.com> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
...ensource_libexec_InterfaceReconfigureVswitch.py | 39 ++++++++++++--------
1 files changed, 23 insertions(+), 16 deletions(-)
Ben Pfaff [Thu, 8 Jul 2010 13:25:51 +0000 (14:25 +0100)]
[PATCH] xenserver: Always call ovs-vsctl in bring_down().
From 56cac225ae4df8ff0c50a806b78abdb6ac96c33a Mon Sep 17 00:00:00 2001
Date: Mon, 22 Feb 2010 13:41:19 -0800
The datapath_modify_config() call here was at the wrong level of
indentation, so it was only getting called if 'dp' was to be deleted.
Signed-off-by: Ben Pfaff <blp@nicira.com> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
...ensource_libexec_InterfaceReconfigureVswitch.py | 3 ++-
1 files changed, 2 insertions(+), 1 deletions(-)
Ben Pfaff [Thu, 8 Jul 2010 13:25:51 +0000 (14:25 +0100)]
[PATCH] xenserver: Fix bringing up VLAN PIFs.
From d77ffabfbcf9eb04b7e69ff6b34e12f000d314ee Mon Sep 17 00:00:00 2001
Date: Mon, 22 Feb 2010 13:08:10 -0800
In configure_datapath(), bringing up a VLAN PIF's datapath is no different
from bringing up a non-VLAN PIF's datapath, but the code was making
unnecessary distinctions. Also, the test
if parent and datapath:
did not work at all, because there was no variable named 'datapath' in
scope and no obvious source of one.
In preconfigure(), it then becomes necessary to create a VLAN bridge for a
VLAN PIF, in a straightforward way.
Signed-off-by: Ben Pfaff <blp@nicira.com> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
...ensource_libexec_InterfaceReconfigureVswitch.py | 34 ++++++++++---------
1 files changed, 18 insertions(+), 16 deletions(-)
Ben Pfaff [Thu, 8 Jul 2010 13:25:51 +0000 (14:25 +0100)]
[PATCH] xenserver: Hoist identical bridge and vswitch functions into common code.
From 96c7918c4043326bfa0e05b3fc9cbb9547279d40 Mon Sep 17 00:00:00 2001
Date: Tue, 23 Feb 2010 09:47:31 -0800
The previous commit made pif_bridge_name() in the bridge and vswitch
versions of interface-reconfigure functionally identical, so this commit
hoists them into a single common implementation in InterfaceReconfigure.py.
pif_is_bridged() also comes along for the ride because it is also generic
and because it is logically related. Only the bridge code uses it at the
moment.
Ben Pfaff [Thu, 8 Jul 2010 13:25:51 +0000 (14:25 +0100)]
[PATCH] xenserver: Accept VLAN PIFs in pif_bridge_name().
From 6987c81c05f21c579fbe60e9b6693862abbbb762 Mon Sep 17 00:00:00 2001
Date: Mon, 22 Feb 2010 12:53:17 -0800
pif_bridge_name() was written when the vswitch interface-reconfigure was
working with the raw Open vSwitch configuration via ovs-cfg-mod. Then,
it made no sense to ask for the "bridge name" of a VLAN PIF, because a
VLAN PIF didn't have a bridge.
Now, however, the vSwitch interface-reconfigure works with ovs-vsctl, which
provides a thin layer over the OVS configuration that makes VLAN bridges
appear to exist. So now it makes sense to ask for the bridge of a VLAN
PIF, and this commit enables that.
Signed-off-by: Ben Pfaff <blp@nicira.com> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
...ensource_libexec_InterfaceReconfigureVswitch.py | 7 +------
1 files changed, 1 insertions(+), 6 deletions(-)
Ben Pfaff [Thu, 8 Jul 2010 13:25:51 +0000 (14:25 +0100)]
[PATCH] xenserver: Delete ports by interface name.
From 7c79588e006eb28e51ca8b715e22abce0c81f5d4 Mon Sep 17 00:00:00 2001
Date: Mon, 22 Feb 2010 16:36:30 -0800
ovs-vsctl command "--if-exists del-port eth0" does nothing if eth0 is on
a bridge as part of a bond, because the bond's port name is not eth0 but
something else. But interface-reconfigure needs to do that, so this commit
adds that ability to ovs-vsctl and modifies interface-reconfigure to use it.
Ben Pfaff [Thu, 8 Jul 2010 13:25:21 +0000 (14:25 +0100)]
[PATCH] xenserver: Fix comment in interface-reconfigure.
From 3b1acc9955fa9c595f6b9e41999aa40041c7b049 Mon Sep 17 00:00:00 2001
Date: Mon, 22 Feb 2010 12:37:20 -0800
This comment was accurate, but didn't actually say what the function did.
Signed-off-by: Ben Pfaff <blp@nicira.com> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
...ensource_libexec_InterfaceReconfigureVswitch.py | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
Rob Hoes [Thu, 8 Jul 2010 12:01:37 +0000 (13:01 +0100)]
Advertise which network backend is in use in DB
For clients to determine which network backend is in use a key "network_backend" is added to the Host.software_version map on each host. The value of this key can be (same as the contents of /var/xensource/network.conf):
* bridge: the Linux bridging backend is in use;
* openvswitch: the Open vSwitch backend is in use.
Daniel Stodden [Thu, 8 Jul 2010 11:59:56 +0000 (12:59 +0100)]
blktap2: Map Tap type devices to blkback backends.
Make the agent map all Vbd backend types to blkback instances. We keep
the kind=Tap on backend types. But the distinction remains significant
only so VHD snapshotting can pause VBDs where necessary.
Signed-off-by: Daniel Stodden <daniel.stodden@citrix.com>
Daniel Stodden [Thu, 8 Jul 2010 11:59:55 +0000 (12:59 +0100)]
blktap2: Redirect VBD pause events to sm/blktap.py
Delegate blkback uevent handling to sm/blktap2.py, a drop-in
replacment for scripts/block.
Once blktap2 is enabled, the new module additionally manages
pause/resume for all tapdisk2 nodes. A VBD.pause/resume then will
pause/resume the tapdev node accordingly.
Signed-off-by: Daniel Stodden <daniel.stodden@citrix.com>
Daniel Stodden [Thu, 8 Jul 2010 11:59:54 +0000 (12:59 +0100)]
scripts/block: Update and disable blkback queue state callbacks.
Fixed, then disabled for now. The change handler is only left around
for reference, and upstreaming to xen.org, eventually.
The queue state variable in kernel space was eliminated, we now exit
the dispatcher for any of pause/shutdown/disconnect operations.
Instead we cleanly write/remove the kthread-pid before signalling
transitions.
Scripts now can hook into shutdown-done too. This is useful because
the hotplug scripts want time to ack the shutdown change, before the
agent gets to remove the backend. Exit signalling might later aid in
other areas, e.g. for consistent ionice modes.
Signed-off-by: Daniel Stodden <daniel.stodden@citrix.com>
Jon Ludlam [Thu, 8 Jul 2010 11:55:00 +0000 (12:55 +0100)]
When a VBD unplug event is caught by the event thread, it issues a VDI.detach but not a VDI.deactivate. This fixes this behaviour. Also does the same for eject (although no ISO SRs currently use activate/deactivate)
Signed-off-by: Jon Ludlam <Jonathan.Ludlam@eu.citrix.com>
Jon Ludlam [Thu, 8 Jul 2010 11:54:49 +0000 (12:54 +0100)]
If there is any error in syncing the RRDs while shutting down, don't let the exception propagate.
Only bad things can happen if the exception gets out.
Signed-off-by: Jon Ludlam <Jonathan.Ludlam@eu.citrix.com>
Rob Hoes [Thu, 8 Jul 2010 11:53:49 +0000 (12:53 +0100)]
Require lifecycle specification for APIs
Another step in the transition to better lifecycle information in the API.
Each API in the datamodel.ml should have an explicit lifecycle description attached to it (which may be [] for prototypes). Specifying 'in_product_since' is still allowed for backwards compatibility, but should no longer be used!
Ben Pfaff [Fri, 18 Jun 2010 18:46:40 +0000 (11:46 -0700)]
Increase SSL private key from 512 to 1024 bits.
At Nicira, we are experimenting with the idea of converting XAPI SSL keys to SSH keys. In some cases this might be convenient for giving XenServers access to remote resources without distributing a second set of keys.
OpenSSH, however, refuses to accept RSA keys shorter than 768 bits for use in authentication. So this change is necessary, to make XAPI generate keys longer than the current default of 512 bits.
Additionally, RSA says "512-bit keys no longer provide sufficient security for anything more than very short-term security needs"
(http://www.rsa.com/rsalabs/node.asp?id=2218), so this change seems like a good idea in any case.
Increasing the key length makes generating the key at installation time take a bit longer, but the difference is not significant:
on my desktop, "openssl genrsa 512" takes about 10 ms and "openssl genrsa 1024" takes about 100 ms.
David Scott [Fri, 18 Jun 2010 19:54:26 +0000 (20:54 +0100)]
In the xapi RPM post-install actions, use 'chkconfig --add' rather than 'chkconfig ... on' otherwise services are shutdown in the wrong order, causing a slave host to freeze during reboot.
(Thanks to James Bulpin for figuring out what was going on :-)
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
David Scott [Fri, 18 Jun 2010 12:53:21 +0000 (13:53 +0100)]
Add a simple script which monitors the paused domains on a host, checks them against the xapi database, logs anomalies, and optionally destroys the domain if it has been in an error state for longer than a threshold (currently 60s)
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
David Scott [Fri, 18 Jun 2010 12:38:31 +0000 (13:38 +0100)]
CA-34549: make the host chooser use the precomputed memory_overhead for consistency.
Now it should be possible to make a single large VM by:
* subtracting the host memory-overhead from memory-total
* subtracting dom0's memory-actual and memory-overhead
* subtracting the overhead of a new VM
* make the new VM use all remaining memory
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
David Scott [Fri, 18 Jun 2010 12:38:30 +0000 (13:38 +0100)]
CA-40530: bump up the threshold time before a host is declared offline to 10 minutes. This is in response to several stress test failures where the heartbeats were delayed by 3 minutes.
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
David Scott [Fri, 18 Jun 2010 12:38:30 +0000 (13:38 +0100)]
CA-40530: Unset the Task.stunnel_pid when the stunnel connections are closed. This prevents the killing of old pids in the event of (i) a task leak; and (ii) a host being declared offline.
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
David Scott [Fri, 18 Jun 2010 12:38:27 +0000 (13:38 +0100)]
CA-41230: In VM.{snapshot,clone}, keep track of whether a new VDI has actually been created and therefore, whether that VDI should be deleted on failure. In particular CDs are shared not duplicated and so the cleanup code shouldn't try to delete them.
Unfortunately the 'writable ISO SR' support changes the default NFS ISO SR mount options to read/write from read/only, exposing this bug.
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Rob Hoes [Fri, 18 Jun 2010 12:38:23 +0000 (13:38 +0100)]
Remove "dechainify VLAN" code from dbsync_slave
This is no longer needed, as the creation of VLANs-on-VLANs is not possible anymore (for about two years), and this function therefore causes unnecessary overhead in xapi's startup sequence. Removed an unused function from the same module as well.
Daniel Stodden [Fri, 18 Jun 2010 12:37:42 +0000 (13:37 +0100)]
Update scripts/block for blkback's new pause/resume ops.
The kernel can emit uevents when quiescing a VBD's I/O queue. It won't
write pause-done, but signal state changes through a new key:
queue-state. The basic idea is to let storage-level code hook into
pause/resume transitions where desirable. This script implements a
default handler.
The original device.ml protocol remains as is, but the implementation
differs:
David Scott [Fri, 18 Jun 2010 12:37:34 +0000 (13:37 +0100)]
CA-41839: Moving to RPMs for xapi had the side-effect of renaming the CLI RPM from xe-cli-... to xapi-xe... This patch changes it back for now to minimise unwanted churn.
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
David Scott [Fri, 18 Jun 2010 12:53:21 +0000 (13:53 +0100)]
Add a simple script which monitors the paused domains on a host, checks them against the xapi database, logs anomalies, and optionally destroys the domain if it has been in an error state for longer than a threshold (currently 60s)
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
diff -r f15a34996e19 scripts/examples/python/monitor-unwanted-domains.py--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/scripts/examples/python/monitor-unwanted-domains.py Fri Jun 04 18:11:13 2010 +0100
@@ -0,0 +1,86 @@
+#!/usr/bin/env python
+
+import subprocess, XenAPI, inventory, time, sys
+
+# Script which monitors the domains running on a host, looks for
+# paused domains which don't correspond to VMs which are running here
+# or are about to run here, logs them and optionally destroys them.
+
+# Return a list of (domid, uuid) tuples, one per paused domain on this host
+def list_paused_domains():
+ results = []
+ all = subprocess.Popen(["/opt/xensource/bin/list_domains"], stdout=subprocess.PIPE).communicate()[0]
+ lines = all.split("\n")
+ for domain in lines[1:]:
+ bits = domain.split()
+ if bits <> []:
+ domid = bits[0]
+ uuid = bits[2]
+ state = bits[4]
+ if 'P' in state:
+ results.append( (domid, uuid) )
+ return results
+
+# Given localhost's uuid and a (domid, uuid) tuple, return True if the domain
+# be somewhere else i.e. we think it may have leaked here
+def should_domain_be_somewhere_else(localhost_uuid, (domid, uuid)):
+ try:
+ x = XenAPI.xapi_local()
+ x.xenapi.login_with_password("root", "")
+ try:
+ try:
+ vm = x.xenapi.VM.get_by_uuid(uuid)
+ resident_on = x.xenapi.VM.get_resident_on(vm)
+ current_operations = x.xenapi.VM.get_current_operations(vm)
+ result = current_operations == {} and resident_on <> localhost_uuid
+ if result:
+ log("domid %s uuid %s: is not being operated on and is not resident here" % (domid, uuid))
+ return result
+ except XenAPI.Failure, e:
+ if e.details[0] == "UUID_INVALID":
+ # VM is totally bogus
+ log("domid %s uuid %s: is not in the xapi database" % (domid, uuid))
+ return True
+ # fail safe for now
+ return False
+ finally:
+ x.xenapi.logout()
+ except:
+ return False
+
+def log(str):
+ print str
+
+# Destroy the given domain
+def destroy_domain((domid, uuid)):
+ log("destroying domid %s uuid %s" % (domid, uuid))
+ all = subprocess.Popen(["/opt/xensource/debug/destroy_domain", "-domid", domid], stdout=subprocess.PIPE).communicate()[0]
+
+# Keep track of when a domain first looked like it should be here
+domain_first_noticed = {}
+
+# Number of seconds after which we conclude that a domain really shouldn't be here
+threshold = 60
+
+if __name__ == "__main__":
+ localhost_uuid = inventory.get_localhost_uuid ()
+ while True:
+ time.sleep(1)
+ paused = list_paused_domains ()
+ # GC the domain_first_noticed map
+ for d in domain_first_noticed.keys():
+ if d not in paused:
+ log("domid %s uuid %s: looks ok now, forgetting about it" % d)
+ del domain_first_noticed[d]
+
+ for d in list_paused_domains():
+ if should_domain_be_somewhere_else(localhost_uuid, d):
+ if d not in domain_first_noticed:
+ domain_first_noticed[d] = time.time()
+ noticed_for = time.time() - domain_first_noticed[d]
+ if noticed_for > threshold:
+ log("domid %s uuid %s: has been in bad state for over threshold" % d)
+ if "-destroy" in sys.argv:
+ destroy_domain(d)
+
+
Magnus Therning [Fri, 18 Jun 2010 12:53:21 +0000 (13:53 +0100)]
[CA-40987]: Fix improper Secret log filtering in the CLI.
Signed-off-by: Magnus Therning <magnus.therning@citrix.com>
diff -r 72b87539d3df ocaml/xapi/xapi_cli.ml--- a/ocaml/xapi/xapi_cli.ml Tue May 04 12:25:34 2010 +0100
+++ b/ocaml/xapi/xapi_cli.ml Wed May 05 16:23:20 2010 +0100
@@ -154,8 +154,13 @@
let rpc = Helpers.get_rpc () req s in
Cli_frontend.populate_cmdtable rpc Ref.null;
(* Log the actual CLI command to help diagnose failures like CA-25516 *)
- debug "xe %s %s" (get_cmdname cmd) (String.concat " " (List.map (fun (k, v) -> let v' = if k = "password" then "(omitted)" else v in k ^ "=" ^ v') params));
- if get_cmdname cmd = "help"
+ let cmd_name = get_cmdname cmd in
+ if String.startswith "secret-" cmd_name
+ then
+ debug "xe %s %s" cmd_name (String.concat " " (List.map (fun (k, v) -> let v' = if k = "value" then "(omitted)" else v in k ^ "=" ^ v') params))
+ else
+ debug "xe %s %s" cmd_name (String.concat " " (List.map (fun (k, v) -> k ^ "=" ^ v) params));
+ if cmd_name = "help"
then do_help is_compat cmd minimal s
else do_rpcs req s u p minimal is_compat cmd session args
David Scott [Fri, 18 Jun 2010 12:38:31 +0000 (13:38 +0100)]
CA-34549: make the host chooser use the precomputed memory_overhead for consistency.
Now it should be possible to make a single large VM by:
* subtracting the host memory-overhead from memory-total
* subtracting dom0's memory-actual and memory-overhead
* subtracting the overhead of a new VM
* make the new VM use all remaining memory
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
diff -r 97206bef788a ocaml/xapi/memory_check.ml--- a/ocaml/xapi/memory_check.ml Mon Mar 29 20:35:46 2010 +0100
+++ b/ocaml/xapi/memory_check.ml Mon Mar 29 21:06:46 2010 +0100
@@ -85,7 +85,7 @@
operation, this function returns the maximum amount of memory that the VM will
need between now, and the point in future time when the operation completes. *)
let vm_compute_used_memory ~__context policy vm_ref =
- if Xapi_fist.disable_memory_checks () then (0L, 0L) else
+ if Xapi_fist.disable_memory_checks () then 0L else
let vm_main_record = Db.VM.get_record ~__context ~self:vm_ref in
let vm_boot_record = Helpers.get_boot_record ~__context ~self:vm_ref in
let memory_static_max = vm_boot_record.API.vM_memory_static_max in
@@ -96,8 +96,7 @@
if ballooning_enabled && policy = Dynamic_min
then memory_dynamic_min
else memory_static_max in
- vm_compute_required_memory vm_boot_record
- (Memory.kib_of_bytes_used memory_required)
+ memory_required +++ vm_main_record.API.vM_memory_overhead
let vm_compute_resume_memory ~__context vm_ref =
if Xapi_fist.disable_memory_checks () then 0L else
@@ -176,8 +175,7 @@
let all_vms = summary.resident @ summary.scheduled in
let all_vm_memories = List.map (vm_compute_used_memory ~__context policy)
all_vms in
- let total_vm_memory = List.fold_left Int64.add 0L
- (List.map (fun (x, y) -> Int64.add x y) all_vm_memories) in
+ let total_vm_memory = List.fold_left Int64.add 0L all_vm_memories in
let host_mem_available = Int64.sub
summary.host_maximum_guest_memory_bytes total_vm_memory in
max 0L host_mem_available
@@ -226,16 +224,14 @@
(mib summary.host_maximum_guest_memory_bytes);
List.iter
(fun v ->
- let main, shadow = vm_compute_used_memory ~__context
- Static_max v in
- debug "Memory_check: VM %s (%s): main memory %Ld (%Ld MiB); \
- shadow memory %Ld (%Ld MiB)"
+ let reqd = vm_compute_used_memory ~__context Static_max v in
+ debug "Memory_check: VM %s (%s): memory %Ld (%Ld MiB)"
(Db.VM.get_uuid ~__context ~self:v)
(if List.mem v summary.resident
then "resident here"
else "scheduled to be resident here"
)
- main (mib main) shadow (mib shadow)
+ reqd (mib reqd)
)
(summary.scheduled @ summary.resident);
debug "Memory_check: available memory: %Ld (%Ld MiB)"
David Scott [Fri, 18 Jun 2010 12:38:30 +0000 (13:38 +0100)]
CA-40530: bump up the threshold time before a host is declared offline to 10 minutes. This is in response to several stress test failures where the heartbeats were delayed by 3 minutes.
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
diff -r 95506f7296fe ocaml/xapi/xapi_globs.ml--- a/ocaml/xapi/xapi_globs.ml Mon Apr 19 20:56:46 2010 +0100
+++ b/ocaml/xapi/xapi_globs.ml Mon Apr 19 20:57:57 2010 +0100
@@ -86,7 +86,7 @@
(* Interval between host heartbeats *)
let host_heartbeat_interval = 30.0
(* If we haven't heard a heartbeat from a host for this interval then the host is assumed dead *)
-let host_assumed_dead_interval = 200.0
+let host_assumed_dead_interval = 600.0 (* 10 minutes *)
David Scott [Fri, 18 Jun 2010 12:38:30 +0000 (13:38 +0100)]
CA-40530: Unset the Task.stunnel_pid when the stunnel connections are closed. This prevents the killing of old pids in the event of (i) a task leak; and (ii) a host being declared offline.
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
diff -r ad754527bddd ocaml/idl/ocaml_backend/xmlrpcclient.ml--- a/ocaml/idl/ocaml_backend/xmlrpcclient.ml Fri Apr 16 11:44:32 2010 +0100
+++ b/ocaml/idl/ocaml_backend/xmlrpcclient.ml Mon Apr 19 20:51:32 2010 +0100
@@ -19,6 +19,7 @@
open D
let set_stunnelpid_callback : (string -> int -> unit) option ref = ref None
+let unset_stunnelpid_callback : (string -> int -> unit) option ref = ref None
(* Headers for an HTTP CONNECT operation *)
let connect_headers ?session_id ?task_id ?subtask_of host path =
@@ -340,14 +341,23 @@
let s = st_proc.Stunnel.fd in
let s_pid = Stunnel.getpid st_proc.Stunnel.pid in
info "stunnel pid: %d (cached = %b) connected to %s:%d" s_pid use_stunnel_cache host port;
- begin
- match task_id with
- None -> debug "Did not write stunnel pid: no task passed to http_rpc fn"
- | Some t ->
- match !set_stunnelpid_callback with
- None -> warn "Did not write stunnel pid: no callback registered"
- | Some f -> f t s_pid
- end;
+
+ (* Call the {,un}set_stunnelpid_callback hooks around the remote call *)
+ let with_recorded_stunnelpid task_opt s_pid f =
+ begin
+ match task_id, !set_stunnelpid_callback with
+ | Some t, Some f -> f t s_pid
+ | _, _ -> ()
+ end;
+ finally f
+ (fun () ->
+ match task_id, !unset_stunnelpid_callback with
+ | Some t, Some f -> f t s_pid
+ | _, _ -> ()
+ ) in
+
+ with_recorded_stunnelpid task_id s_pid
+ (fun () ->
finally
(fun () ->
try
@@ -368,7 +378,7 @@
Stunnel.disconnect st_proc
end
)
-
+ )
(** Take an optional content_length and task_id together with a socket
and return the XMLRPC response as an XML document *)
David Scott [Fri, 18 Jun 2010 12:38:27 +0000 (13:38 +0100)]
CA-41230: In VM.{snapshot,clone}, keep track of whether a new VDI has actually been created and therefore, whether that VDI should be deleted on failure. In particular CDs are shared not duplicated and so the cleanup code shouldn't try to delete them.
Unfortunately the 'writable ISO SR' support changes the default NFS ISO SR mount options to read/write from read/only, exposing this bug.
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
diff -r 2a39d3f3842e ocaml/xapi/xapi_vm_clone.ml--- a/ocaml/xapi/xapi_vm_clone.ml Mon May 10 17:14:10 2010 +0100
+++ b/ocaml/xapi/xapi_vm_clone.ml Mon May 10 17:14:17 2010 +0100
@@ -21,7 +21,11 @@
open D
let delete_disks rpc session_id disks =
- List.iter (fun (vbd,vdi) -> try Client.VDI.destroy rpc session_id vdi with _ -> ()) disks
+ List.iter (fun (vbd,vdi,on_error_delete) ->
+ if on_error_delete
+ then try Client.VDI.destroy rpc session_id vdi with _ -> ()
+ else debug "Not destroying CD VDI: %s" (Ref.string_of vdi)
+ ) disks
let wait_for_clone ?progress_minmax ~__context task =
Helpers.call_api_functions ~__context (fun rpc session ->
@@ -134,14 +138,14 @@
(* If the VBD is empty there is no VDI to copy. *)
(* If the VBD is a CD then eject it (we cannot make copies of ISOs: they're identified *)
(* by their filename unlike other VDIs) *)
- let newvdi =
+ let newvdi, on_error_delete =
if vbd_r.API.vBD_empty
- then Ref.null
+ then Ref.null, false
else if vbd_r.API.vBD_type = `CD
- then vbd_r.API.vBD_VDI
- else clone_single_vdi ~progress:(done_so_far, size, total) rpc session_id disk_op ~__context vbd_r.API.vBD_VDI driver_params
+ then vbd_r.API.vBD_VDI, false (* don't delete the original CD *)
+ else clone_single_vdi ~progress:(done_so_far, size, total) rpc session_id disk_op ~__context vbd_r.API.vBD_VDI driver_params, true (* do delete newly created VDI *)
in
- ((vbd,newvdi)::acc, (Int64.add done_so_far size))
+ ((vbd,newvdi,on_error_delete)::acc, (Int64.add done_so_far size))
with e ->
debug "Error in safe_clone_disks: %s" (Printexc.to_string e);
delete_disks rpc session_id acc; (* Delete those cloned so far *)
@@ -348,7 +352,7 @@
(* copy VBDs *)
let new_vbds : [`VBD] Ref.t list =
- List.map (fun (vbd, newvdi) -> Xapi_vbd_helpers.copy ~__context ~vm:ref ~vdi:newvdi vbd) cloned_disks in
+ List.map (fun (vbd, newvdi, _) -> Xapi_vbd_helpers.copy ~__context ~vm:ref ~vdi:newvdi vbd) cloned_disks in
(* copy VIFs *)
let new_vifs : [`VIF] Ref.t list =
Rob Hoes [Fri, 18 Jun 2010 12:38:23 +0000 (13:38 +0100)]
Remove "dechainify VLAN" code from dbsync_slave
This is no longer needed, as the creation of VLANs-on-VLANs is not possible anymore (for about two years), and this function therefore causes unnecessary overhead in xapi's startup sequence. Removed an unused function from the same module as well.
Signed-off-by: Rob Hoes <rob.hoes@citrix.com>
diff -r 7e09c13b96e7 ocaml/xapi/dbsync_slave.ml--- a/ocaml/xapi/dbsync_slave.ml
+++ b/ocaml/xapi/dbsync_slave.ml
@@ -31,14 +31,6 @@
let ( ** ) = Int64.mul
let ( // ) = Int64.div
-let trim_end s =
- let i = ref (String.length s - 1) in
- while !i > 0 && (List.mem s.[!i] [ ' '; '\t'; '\n'; '\r' ])
- do
- decr i
- done;
- if !i >= 0 then String.sub s 0 (!i + 1) else ""
-
(* create localhost record *)
let get_my_ip_addr() =
@@ -115,35 +107,6 @@
end else
Db.Host.remove_from_other_config ~__context ~self:host ~key:Xapi_globs.host_no_local_storage
-(* CA-25162: Dechainify VLANs. We're actually doing this for _all_
- * PIFs, not just those relevant to localhost. Mostly this will be
- * a no-op, and it shouldn't matter if we fix problems for other hosts
- * here, and it covers the case where we're a slave and the master has
- * broken vlans which need to be corrected before we try to replicate
- * them *)
-let fix_chained_vlans ~__context =
- let pifs = Db.PIF.get_all_records ~__context in
- let (vlan_pifs,underlying_pifs) = List.partition (fun (_,pifr) -> pifr.API.pIF_VLAN >= 0L) pifs in
- List.iter (fun (vlan_pif_ref,vlan_pif_record) ->
- let pif_underneath_vlan = Helpers.get_pif_underneath_vlan ~__context vlan_pif_ref in
- if not (List.exists (fun (pif_ref,_) -> pif_ref = pif_underneath_vlan) underlying_pifs) then begin
- (* There's a problem - the underlying PIF of the vlan might be a vlan itself (or might not exist)
- Find the real underlying PIF by matching the host and device *)
- try
- let (real_pif_ref,real_pif_rec) = List.find (fun (_,pif_rec) ->
- pif_rec.API.pIF_host = vlan_pif_record.API.pIF_host &&
- pif_rec.API.pIF_device = vlan_pif_record.API.pIF_device) underlying_pifs in
- let vlan = Db.PIF.get_VLAN_master_of ~__context ~self:vlan_pif_ref in
- warn "Resetting tagged PIF of VLAN %s, previously was %s" (Ref.string_of vlan) (Ref.string_of pif_underneath_vlan);
- Db.VLAN.set_tagged_PIF ~__context ~self:vlan ~value:real_pif_ref
- with _ ->
- (* Can't find an underlying PIF - delete the VLAN record. This is pretty unlikely. *)
- error "Destroying dangling VLAN and associated PIF record - the underlying device has disappeared";
- let vlan = Db.PIF.get_VLAN_master_of ~__context ~self:vlan_pif_ref in
- Db.VLAN.destroy ~__context ~self:vlan;
- Db.PIF.destroy ~__context ~self:vlan_pif_ref
- end) vlan_pifs
-
(*************** update database tools ******************)
Rob Hoes [Fri, 18 Jun 2010 12:38:15 +0000 (13:38 +0100)]
Require lifecycle specification for APIs
Another step in the transition to better lifecycle information in the API.
Each API in the datamodel.ml should have an explicit lifecycle description attached to it (which may be [] for prototypes). Specifying 'in_product_since' is still allowed for backwards compatibility, but should no longer be used!
-let call ~name ?(doc="") ?(in_oss_since=Some "3.0.3") ~in_product_since ?internal_deprecated_since
+let get_published lifecycle =
+ try
+ let _, published, _ = List.find (fun (t, _, _) -> t = Published) lifecycle in
+ Some published
+ with Not_found -> None
+
+let get_deprecated lifecycle =
+ try
+ let _, deprecated, _ = List.find (fun (t, _, _) -> t = Deprecated) lifecycle in
+ Some deprecated
+ with Not_found -> None
+
+let call ~name ?(doc="") ?(in_oss_since=Some "3.0.3") ?in_product_since ?internal_deprecated_since
?result ?(flags=[`Session;`Async])
?(effect=true) ?(tag=Custom) ?(errs=[]) ?(custom_marshaller=false) ?(db_only=false)
?(no_current_operations=false) ?(secret=false) ?(hide_from_docs=false)
@@ -196,20 +208,30 @@
(* if you specify versioned_params then these get put in the params field of the message record;
* otherwise params go in with no default values and param_release=call_release...
*)
- let call_release = {internal=get_product_releases in_product_since;
- opensource=get_oss_releases in_oss_since;
- internal_deprecated_since = internal_deprecated_since;
- } in
+ if lifecycle = None && in_product_since = None then
+ failwith ("Lifecycle for message '" ^ name ^ "' not specified");
let lifecycle = match lifecycle with
| None ->
- let publish = [Published, in_product_since, doc] in
+ let published = match in_product_since with
+ | None -> []
+ | Some rel -> [Published, rel, doc]
+ in
let deprecated = match internal_deprecated_since with
| None -> []
| Some rel -> [Deprecated, rel, ""]
in
- publish @ deprecated
+ published @ deprecated
| Some l -> l
in
+ let call_release =
+ {
+ internal = (match get_published lifecycle with
+ | Some published -> get_product_releases published
+ | None -> ["closed"]);
+ opensource = get_oss_releases in_oss_since;
+ internal_deprecated_since = get_deprecated lifecycle;
+ }
+ in
{
msg_name = name;
msg_params =
@@ -2637,28 +2659,40 @@
(** Make an object field record *)
-let field ?(in_oss_since = Some "3.0.3") ?(in_product_since = rel_rio) ?(internal_only = false)
+let field ?(in_oss_since = Some "3.0.3") ?in_product_since ?(internal_only = false)
?internal_deprecated_since ?(ignore_foreign_key = false) ?(writer_roles=None) ?(reader_roles=None)
?(qualifier = RW) ?(ty = String) ?(effect = false) ?(default_value = None) ?(persist = true)
?(map_keys_roles=[]) (* list of (key_name,(writer_roles)) for a map field *)
?lifecycle name desc =
-
+ (* in_product_since currently defaults to 'Some rel_rio', for backwards compatibility.
+ * This should eventually become 'None'. *)
+ let in_product_since = match in_product_since with None -> Some rel_rio | x -> x in
+ if lifecycle = None && in_product_since = None then
+ failwith ("Lifecycle for field '" ^ name ^ "' not specified");
let lifecycle = match lifecycle with
- | None ->
- let publish = [Published, in_product_since, desc] in
- let deprecated = match internal_deprecated_since with
- | None -> []
- | Some rel -> [Deprecated, rel, ""]
- in
- publish @ deprecated
- | Some l -> l
+ | None ->
+ let published = match in_product_since with
+ | None -> []
+ | Some rel -> [Published, rel, desc]
+ in
+ let deprecated = match internal_deprecated_since with
+ | None -> []
+ | Some rel -> [Deprecated, rel, ""]
+ in
+ published @ deprecated
+ | Some l -> l
+ in
+ let release =
+ {
+ internal = (match get_published lifecycle with
+ | Some published -> get_product_releases published
+ | None -> ["closed"]);
+ opensource = get_oss_releases in_oss_since;
+ internal_deprecated_since = get_deprecated lifecycle;
+ }
in
Field {
- release = {
- internal=get_product_releases in_product_since;
- opensource=(get_oss_releases in_oss_since);
- internal_deprecated_since=internal_deprecated_since;
- };
+ release = release;
lifecycle=lifecycle;
qualifier=qualifier; ty=ty; internal_only = internal_only; default_value = default_value;
field_name=name;
@@ -2704,7 +2738,7 @@
let default_field_writer_roles = _R_POOL_ADMIN (* by default, only root can write to them *)
(** Create an object and map the object name into the messages *)
-let create_obj ?lifecycle ~in_oss_since ~in_product_since ~internal_deprecated_since ~gen_constructor_destructor ~gen_events ~persist ~name ~descr ~doccomments ~contents ~messages ~in_db
+let create_obj ?lifecycle ~in_oss_since ?in_product_since ?(internal_deprecated_since=None) ~gen_constructor_destructor ~gen_events ~persist ~name ~descr ~doccomments ~contents ~messages ~in_db
?(contents_default_reader_roles=default_field_reader_roles) ?(contents_default_writer_roles=None)
?(implicit_messages_allowed_roles=_R_ALL) (* used in implicit obj msgs (get_all, etc) *)
?force_custom_actions:(force_custom_actions=None) (* None,Some(RW),Some(StaticRO) *)
@@ -2719,20 +2753,34 @@
| Field f->Field{f with field_setter_roles=get_field_writer_roles f.field_setter_roles;
field_getter_roles=get_field_reader_roles f.field_getter_roles}
) contents in
+ if lifecycle = None && in_product_since = None then
+ failwith ("Lifecycle for class '" ^ name ^ "' not specified");
let lifecycle = match lifecycle with
| None ->
- let publish = [Published, in_product_since, descr] in
+ let published = match in_product_since with
+ | None -> []
+ | Some rel -> [Published, rel, descr]
+ in
let deprecated = match internal_deprecated_since with
| None -> []
| Some rel -> [Deprecated, rel, ""]
in
- publish @ deprecated
+ published @ deprecated
| Some l -> l
+ in
+ let release =
+ {
+ internal = (match get_published lifecycle with
+ | Some published -> get_product_releases published
+ | None -> ["closed"]);
+ opensource = get_oss_releases in_oss_since;
+ internal_deprecated_since = get_deprecated lifecycle;
+ }
in
let msgs = List.map (fun m -> {m with msg_obj_name=name;msg_allowed_roles=get_msg_allowed_roles m.msg_allowed_roles}) messages in
{ name = name; description = descr; obj_lifecycle = lifecycle; messages = msgs; contents = contents;
doccomments = doccomments; gen_constructor_destructor = gen_constructor_destructor; force_custom_actions = force_custom_actions;
- persist = persist; gen_events = gen_events; obj_release = {internal=get_product_releases in_product_since; opensource=get_oss_releases in_oss_since; internal_deprecated_since = internal_deprecated_since};
+ persist = persist; gen_events = gen_events; obj_release = release;
in_database=in_db; obj_allowed_roles = messages_default_allowed_roles; obj_implicit_msg_allowed_roles = implicit_messages_allowed_roles;
}
%% Document authors
\newcommand{\docauthors}{
}
-\newcommand{\legalnotice}{Copyright \copyright{} 2006-2008 Citrix Systems, Inc. All Rights Reserved.}
+\newcommand{\legalnotice}{Copyright \copyright{} 2006-2010 Citrix Systems, Inc. All Rights Reserved.}
Daniel Stodden [Fri, 18 Jun 2010 12:37:42 +0000 (13:37 +0100)]
Update scripts/block for blkback's new pause/resume ops.
The kernel can emit uevents when quiescing a VBD's I/O queue. It won't
write pause-done, but signal state changes through a new key:
queue-state. The basic idea is to let storage-level code hook into
pause/resume transitions where desirable. This script implements a
default handler.
The original device.ml protocol remains as is, but the implementation
differs:
David Scott [Fri, 18 Jun 2010 12:37:34 +0000 (13:37 +0100)]
CA-41839: Moving to RPMs for xapi had the side-effect of renaming the CLI RPM from xe-cli-... to xapi-xe... This patch changes it back for now to minimise unwanted churn.
Rob Hoes [Thu, 17 Jun 2010 16:31:00 +0000 (17:31 +0100)]
Remove bond-slave filter from PIF.scan
This filter uses sysfs to find out whether an interface is a bond slave. This does not work when the openvswitch backend is used. Besides, it does not seem to be necessary (anymore?) anyway.
PIF.scan creates a new PIF for each interface it finds if this interface is physical and there is no PIF with the same MAC. The slave interfaces always have the same MAC as the bond master PIF (bridging backend) or their "real" MACs as stored in their PIFs (openvswitch backend). Therefore, when a bond is in effect, the MACs of all bond slaves have corresponding PIFs in the DB, so PIF.scan won't create new ones.
Rob Hoes [Thu, 17 Jun 2010 16:31:00 +0000 (17:31 +0100)]
CA-40910: Only copy physical PIFs on pool join
On pool join, any bonds or VLANs on the joining host are supposed to be ignored. When the host reboots after the join, it will inherit the bond/VLAN setup from the pool master. Therefore, Bond and VLAN objects are not copied from the joining host to the pool. However, bond and VLAN master PIFs are, with is not good. This patch fixes that.
Rob Hoes [Thu, 17 Jun 2010 16:31:00 +0000 (17:31 +0100)]
Split off bond/VLAN reconstruction code from Dbsync_slave
When a pool slave starts up, it needs synchronise its bonds and VLANs with the pool master. This code is in the Dbsync_slave module, but does not really belong there (DBsync_slave is for synchronising DB fields such as PIF.currently_attached with the state of the physical system). This patch gives the bond/VLAN recreation code its own module.
Jonathan Knowles [Wed, 19 May 2010 10:50:11 +0000 (11:50 +0100)]
[CA-41286] Fixes a memory accounting error in the squeezing code.
When calculating an upper bound for the memory usage of an HVM domain with no balloon driver, the squeezing code would double-count (inappropriately) the shadow memory area.
This caused the squeezing algorithm to underestimate the amount of usable memory of any host running one or more HVM domains without balloon drivers.
This change removes the double-counting error, and also fixes a bug that caused the squeezer to add (inappropriately) the shadow memory size to Xen maxmem.
Signed-off-by: Jonathan Knowles <jonathan.knowles@eu.citrix.com> Acked-by: Dave Scott <dave.scott@eu.citrix.com>
David Scott [Thu, 13 May 2010 13:31:01 +0000 (14:31 +0100)]
Revert c/s 848:dddf116cddb5 - VM.other_config:auto_poweron is inexplicable either way around, so revert back to the old behaviour until we come up with a new plan (probably involving merging this with HA VM restart)
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
Rob Hoes [Thu, 17 Jun 2010 16:31:00 +0000 (17:31 +0100)]
Updated API evolution document
This document is meant to describe how the elements of the XenAPI may evolve over time, and provide compatibility guidelines. This is only a proposal and is still incomplete; we need some discussion about it, especially to establish what sort of compatibility guarantees we can give.
Zheng Li [Tue, 4 May 2010 05:22:08 +0000 (06:22 +0100)]
A few bugfixes and a few minor improvements to current xe cmdline tool implementation
Bugfixes:
* xe command line options doesn't mix well with XE_EXTRA_ARGS varialbe. E.g. setting XE_EXTRA_ARGS to "username=xxxx,password=yyyy" (or any non-nil valid configuration) and calling "xe -s <some server> vm-list" will break xe. Note that this is a common user case in a cluster-like environment where all the machines have the same user/passwd config, where one can conveniently set user/passwd in XE_EXTRA_ARGS for once and connect to different servers by only specifying different "-s" arguments in the cmdline.
* Setting "compat=true" in xe's RC file won't work. E.g. xe vm-clone vm-name=<vm name> new-name=<new vm name> with "compat=true" in ~/.xe won't work (but with "compat=true" in XE_EXTRA_ARGS or in xe cmdline will work).
* Setting a password with comma via XE_EXTRA_ARGS will break the logic. After the fix, it's possible to specify that by using backslash to escape the comma (e.g. password=pass\,word)
* clean up the options handling logic, so that cmdline options, RC file setting and XE_EXTRA_ARGS variable can mix consistently even in some corner cases and follow the natural priority: cmdline option > XE_EXTRA_ARGS > ~/.xe RC > default settings
Improvements:
* change options "-debug" and "-debug-on-fail" to "--debug" and "--debug-on-fail", so that every command line option now follows the common naming convention of -shortcut v.s.--full-name (with the only standard exception of having both "-help" and "--help"). AFAICS, both debug options are (maybe deliberately) not documented in the manual, so changing the names might not be a big issue regarding compatibilities.
* complete the pair relation between command line options and RC/environment variables. There were some missings from either side: e.g. "compat=xxxx" has no "--compat" correspondence and "--debug"("--debug-on-fail") has not "debug=xxxx" in par.
Zheng Li [Fri, 23 Apr 2010 18:30:04 +0000 (19:30 +0100)]
Add missing edition "XD"
"XD" edition seems to be missing here. The more interesting thing is that when we fail because of this bug, the log file will say "unknown edition" as expected, but the cli's error message will say "daemon was not found".
Zheng Li [Fri, 23 Apr 2010 17:54:31 +0000 (18:54 +0100)]
Enable native/bytecode compilation choices, and some Makefile simplification.
* Enable the independent bytecode compilation mode.
Previously this was unfortunately blocked by the hard wired xapi-client.cmxa in the Makefile. Now we should be able to load the bytecode version of xapi client lib in the OCaml interpreter for interactive development with instant feedback. Use $(COMPILE_NATIVE) and $(COMPILE_BYTE) to control these options.
* Change a few "XXX_TARGET = $(if $(equal $(XXX_CONF), yes), $(XXX_OBJ), none)" to "XXX_TARGET = $(if $(XXX_CONF), $(XXX_OBJ))"
The judge conditions are not strictly equivalent in semantics. The former holds only if XXX_CONF equals to "yes", the later condition holds when XXX_CONF is set to any non-negative string (not "no", "false", "0", none etc.). But I think it should be the same regarding XAPI's current Makefiles, and with more simplicity and tolerance. A lot more simplication and optimization can be done to these Makefiles, it's just a matter of time. Moreover XAPI should consider to update to new version of OMake (which should be better, but unfortunately not fully compatible with the old one).
David Scott [Fri, 9 Apr 2010 18:57:43 +0000 (19:57 +0100)]
CA-39291: Work around firewalls which kill idle TCP connections by inserting a small empty block into an export every 5s or so.
The failure happens whenever a disk has a lot of zeroes in it: the TCP connection goes idle while the server is scanning for the next non-zero block. Even setting SO_KEEPALIVE on the stunnel sockets and reducing the window probe interval down to 30s didn't fix it. We wish to keep the ability to have a basic client do an export via HTTP GET so we can't add application-level keepalives to the protocol... we must add them to the export itself.
Note this change is backwards compatible. The receiver code expects:
* a common prefix
* a monotonically increasing chunk number
* the first and last blocks to be the same size and included verbatim (even if all zeroes)
* blocks of zeroes the same size as the first block represented as gaps in the increasing chunk number sequence
Therefore including extra files of length 0 in the stream will be ignored provided they
* share the common prefix and chunk numbering scheme
* are not the first or last blocks
Signed-off-by: David Scott <dave.scott@eu.citrix.com>
projects / xcp / xen-api.git / log