From 575c839b6efa0d093f422688179d8c2b57e8fb2d Mon Sep 17 00:00:00 2001 From: "Daniel P. Berrange" Date: Fri, 13 Feb 2015 17:02:44 +0000 Subject: [PATCH] docs: split out sanlock setup docs In preparation for adding docs about virtlockd, split out the sanlock setup docs into a separate page. Signed-off-by: Daniel P. Berrange --- docs/locking-sanlock.html.in | 247 +++++++++++++++++++++++++++++++ docs/locking.html.in | 277 +++++------------------------------ docs/sitemap.html.in | 6 + 3 files changed, 286 insertions(+), 244 deletions(-) create mode 100644 docs/locking-sanlock.html.in diff --git a/docs/locking-sanlock.html.in b/docs/locking-sanlock.html.in new file mode 100644 index 000000000..84b1fb3f7 --- /dev/null +++ b/docs/locking-sanlock.html.in @@ -0,0 +1,247 @@ + + + + +

Virtual machine lock manager, sanlock plugin

+ +

+ This page describes use of the + sanlock + service as a lock driver + plugin for virtual machine disk mutual exclusion. +

+ +

Sanlock daemon setup

+ +

+ On many operating systems, the sanlock plugin + is distributed in a sub-package which needs to be installed + separately from the main libvirt RPM. On a Fedora/RHEL host + this can be done with the yum command +

+ +

+      $ su - root
+      # yum install libvirt-lock-sanlock
+

+ +

+ The next step is to start the sanlock daemon. For maximum + safety sanlock prefers to have a connection to a watchdog + daemon. This will cause the entire host to be rebooted in + the event that sanlock crashes / terminates abnormally. + To start the watchdog daemon on a Fedora/RHEL host + the following commands can be run: +

+ +

+      $ su - root
+      # chkconfig wdmd on
+      # service wdmd start
+

+ +

+ Once the watchdog is running, sanlock can be started + as follows +

+ +

+      # chkconfig sanlock on
+      # service sanlock start
+

+ +

+ Note: if you wish to avoid the use of the + watchdog, add the following line to /etc/sysconfig/sanlock + before starting it +

+ +

+      SANLOCKOPTS="-w 0"
+

+ +

+ The sanlock daemon must be started on every single host + that will be running virtual machines. So repeat these + steps as necessary. +

+ +

libvirt sanlock plugin configuration

+ +

+ Once the sanlock daemon is running, the next step is to + configure the libvirt sanlock plugin. There is a separate + configuration file for each libvirt driver that is using + sanlock. For QEMU, we will edit /etc/libvirt/qemu-sanlock.conf + There is one mandatory parameter that needs to be set, + the host_id. This is a integer between + 1 and 2000, which must be set to a unique + value on each host running virtual machines. +

+ +

+      $ su - root
+      # augtool -s set /files/etc/libvirt/qemu-sanlock.conf/host_id 1
+

+ +

+ Repeat this on every host, changing 1 to a + unique value for the host. +

+ +

libvirt sanlock storage configuration

+ +

+ The sanlock plugin needs to create leases in a directory + that is on a filesystem shared between all hosts running + virtual machines. Obvious choices for this include NFS + or GFS2. The libvirt sanlock plugin expects its lease + directory be at /var/lib/libvirt/sanlock + so update the host's /etc/fstab to mount + a suitable shared/cluster filesystem at that location +

+ +

+      $ su - root
+      # echo "some.nfs.server:/export/sanlock /var/lib/libvirt/sanlock nfs hard,nointr 0 0" >> /etc/fstab
+      # mount /var/lib/libvirt/sanlock
+

+ +

+ If your sanlock daemon happen to run under non-root + privileges, you need to tell this to libvirt so it + chowns created files correctly. This can be done by + setting user and/or group + variables in the configuration file. Accepted values + range is specified in description to the same + variables in /etc/libvirt/qemu.conf. For + example: +

+ +

+      augtool -s set /files/etc/libvirt/qemu-sanlock.conf/user sanlock
+      augtool -s set /files/etc/libvirt/qemu-sanlock.conf/group sanlock
+

+ +

+ But remember, that if this is NFS share, you need a + no_root_squash-ed one for chown (and chmod possibly) + to succeed. +

+ +

+ In terms of storage requirements, if the filesystem + uses 512 byte sectors, you need to allow for 1MB + of storage for each guest disk. So if you have a network + with 20 virtualization hosts, each running 50 virtual + machines and an average of 2 disks per guest, you will + need 20*50*2 == 2000 MB of storage for + sanlock. +

+ + +

+ On one of the hosts on the network is it wise to setup + a cron job which runs the virt-sanlock-cleanup + script periodically. This scripts deletes any lease + files which are not currently in use by running virtual + machines, freeing up disk space on the shared filesystem. + Unless VM disks are very frequently created + deleted + it should be sufficient to run the cleanup once a week. +

+ +

QEMU/KVM driver configuration

+ +

+ The QEMU/KVM driver is fully integrated with the lock + manager framework as of release 0.9.3. + The out of the box configuration, however, currently + uses the nop lock manager plugin. + To get protection for disks, it is thus necessary + to reconfigure QEMU to activate the sanlock + driver. This is achieved by editing the QEMU driver + configuration file (/etc/libvirt/qemu.conf) + and changing the lock_manager configuration + tunable. +

+ +

+      $ su - root
+      # augtool -s  set /files/etc/libvirt/qemu.conf/lock_manager sanlock
+      # service libvirtd restart
+

+ +

+ If all went well, libvirtd will have talked to sanlock + and created the basic lockspace. This can be checked + by looking for existence of the following file +

+ +

+      # ls /var/lib/libvirt/sanlock/
+      __LIBVIRT__DISKS__
+

+ +

+ Every time you start a guest, additional lease files will appear + in this directory, one for each virtual disk. The lease + files are named based on the MD5 checksum of the fully qualified + path of the virtual disk backing file. So if the guest is given + a disk backed by /var/lib/libvirt/images/demo.img + expect to see a lease /var/lib/libvirt/sanlock/bfa0240911bc17753e0b473688822159 +

+ +

+ It should be obvious that for locking to work correctly, every + host running virtual machines should have storage configured + in the same way. The easiest way to do this is to use the libvirt + storage pool capability to configure any NFS volumes, iSCSI targets, + or SCSI HBAs used for guest storage. Simply replicate the same + storage pool XML across every host. It is important that any + storage pools exposing block devices are configured to create + volume paths under /dev/disks/by-path to ensure + stable paths across hosts. An example iSCSI configuration + which ensures this is: +

+ +

+<pool type='iscsi'>
+  <name>myiscsipool</name>
+  <source>
+    <host name='192.168.254.8'/>
+    <device path='your-iscsi-target-iqn'/>
+  </source>
+  <target>
+    <path>/dev/disk/by-path</path>
+  </target>
+</pool>
+

+ +

Domain configuration

+ +

+ In case sanlock loses access to disk locks for some reason, it will + kill all domains that lost their locks. This default behavior may + be changed using + on_lockfailure + element in domain XML. When this element is present, sanlock + will call sanlock_helper (provided by libvirt) with + the specified action. This helper binary will connect to libvirtd + and thus it may need to authenticate if libvirtd was configured to + require that on the read-write UNIX socket. To provide the + appropriate credentials to sanlock_helper, a + client authentication + file needs to contain something like the following: +

+[auth-libvirt-localhost]
+credentials=sanlock
+
+[credentials-sanlock]
+authname=login
+password=password
+

+ + diff --git a/docs/locking.html.in b/docs/locking.html.in index 8e1e72c73..cf72eb000 100644 --- a/docs/locking.html.in +++ b/docs/locking.html.in @@ -2,258 +2,47 @@ -

Virtual machine disk locking

Virtual machine lock manager

- This page describes how to ensure a single disk cannot be - used by more than one running VM at a time, across any - host in a network. This is critical to avoid data corruption - of guest files systems that are not cluster aware. + Libvirt includes a framework for ensuring mutual exclusion + between virtual machines using host resources. Typically + this is used to prevent two VM processes from having concurrent + write access to the same disk image, as this would result in + data corruption if the guest was not using a cluster + aware filesystem.

Lock manager plugins

- libvirt includes a pluggable framework for lock managers, - which hypervisor drivers can use to ensure safety for - guest domain disks, and potentially other resources. - At this time there are only two plugin implementations, - a "no op" implementation which does absolutely nothing, - and a sanlock implementation which uses - the Disk Paxos algorithm to ensure safety. -

- -

Sanlock daemon setup

- -

- On many operating systems, the sanlock plugin - is distributed in a sub-package which needs to be installed - separately from the main libvirt RPM. On a Fedora/RHEL host - this can be done with the yum command -

- -

-      $ su - root
-      # yum install libvirt-lock-sanlock
-

- -

- The next step is to start the sanlock daemon. For maximum - safety sanlock prefers to have a connection to a watchdog - daemon. This will cause the entire host to be rebooted in - the event that sanlock crashes / terminates abnormally. - To start the watchdog daemon on a Fedora/RHEL host - the following commands can be run: -

- -

-      $ su - root
-      # chkconfig wdmd on
-      # service wdmd start
-

- -

- Once the watchdog is running, sanlock can be started - as follows -

- -

-      # chkconfig sanlock on
-      # service sanlock start
-

- -

- Note: if you wish to avoid the use of the - watchdog, add the following line to /etc/sysconfig/sanlock - before starting it -

- -

-      SANLOCKOPTS="-w 0"
-

- -

- The sanlock daemon must be started on every single host - that will be running virtual machines. So repeat these - steps as necessary. -

- -

libvirt sanlock plugin configuration

- -

- Once the sanlock daemon is running, the next step is to - configure the libvirt sanlock plugin. There is a separate - configuration file for each libvirt driver that is using - sanlock. For QEMU, we will edit /etc/libvirt/qemu-sanlock.conf - There is one mandatory parameter that needs to be set, - the host_id. This is a integer between - 1 and 2000, which must be set to a unique - value on each host running virtual machines. -

- -

-      $ su - root
-      # augtool -s set /files/etc/libvirt/qemu-sanlock.conf/host_id 1
-

- -

- Repeat this on every host, changing 1 to a - unique value for the host. -

- -

libvirt sanlock storage configuration

- -

- The sanlock plugin needs to create leases in a directory - that is on a filesystem shared between all hosts running - virtual machines. Obvious choices for this include NFS - or GFS2. The libvirt sanlock plugin expects its lease - directory be at /var/lib/libvirt/sanlock - so update the host's /etc/fstab to mount - a suitable shared/cluster filesystem at that location -

- -

-      $ su - root
-      # echo "some.nfs.server:/export/sanlock /var/lib/libvirt/sanlock nfs hard,nointr 0 0" >> /etc/fstab
-      # mount /var/lib/libvirt/sanlock
-

- -

- If your sanlock daemon happen to run under non-root - privileges, you need to tell this to libvirt so it - chowns created files correctly. This can be done by - setting user and/or group - variables in the configuration file. Accepted values - range is specified in description to the same - variables in /etc/libvirt/qemu.conf. For - example: -

- -

-      augtool -s set /files/etc/libvirt/qemu-sanlock.conf/user sanlock
-      augtool -s set /files/etc/libvirt/qemu-sanlock.conf/group sanlock
-

- -

- But remember, that if this is NFS share, you need a - no_root_squash-ed one for chown (and chmod possibly) - to succeed. -

- -

- In terms of storage requirements, if the filesystem - uses 512 byte sectors, you need to allow for 1MB - of storage for each guest disk. So if you have a network - with 20 virtualization hosts, each running 50 virtual - machines and an average of 2 disks per guest, you will - need 20*50*2 == 2000 MB of storage for - sanlock. -

- - -

- On one of the hosts on the network is it wise to setup - a cron job which runs the virt-sanlock-cleanup - script periodically. This scripts deletes any lease - files which are not currently in use by running virtual - machines, freeing up disk space on the shared filesystem. - Unless VM disks are very frequently created + deleted - it should be sufficient to run the cleanup once a week. -

- -

QEMU/KVM driver configuration

- -

- The QEMU/KVM driver is fully integrated with the lock - manager framework as of release 0.9.3. - The out of the box configuration, however, currently - uses the nop lock manager plugin. - To get protection for disks, it is thus necessary - to reconfigure QEMU to activate the sanlock - driver. This is achieved by editing the QEMU driver - configuration file (/etc/libvirt/qemu.conf) - and changing the lock_manager configuration - tunable. -

- -

-      $ su - root
-      # augtool -s  set /files/etc/libvirt/qemu.conf/lock_manager sanlock
-      # service libvirtd restart
-

- -

- If all went well, libvirtd will have talked to sanlock - and created the basic lockspace. This can be checked - by looking for existence of the following file -

- -

-      # ls /var/lib/libvirt/sanlock/
-      __LIBVIRT__DISKS__
-

- -

- Every time you start a guest, additional lease files will appear - in this directory, one for each virtual disk. The lease - files are named based on the MD5 checksum of the fully qualified - path of the virtual disk backing file. So if the guest is given - a disk backed by /var/lib/libvirt/images/demo.img - expect to see a lease /var/lib/libvirt/sanlock/bfa0240911bc17753e0b473688822159 -

- -

- It should be obvious that for locking to work correctly, every - host running virtual machines should have storage configured - in the same way. The easiest way to do this is to use the libvirt - storage pool capability to configure any NFS volumes, iSCSI targets, - or SCSI HBAs used for guest storage. Simply replicate the same - storage pool XML across every host. It is important that any - storage pools exposing block devices are configured to create - volume paths under /dev/disks/by-path to ensure - stable paths across hosts. An example iSCSI configuration - which ensures this is: -

- -

-<pool type='iscsi'>
-  <name>myiscsipool</name>
-  <source>
-    <host name='192.168.254.8'/>
-    <device path='your-iscsi-target-iqn'/>
-  </source>
-  <target>
-    <path>/dev/disk/by-path</path>
-  </target>
-</pool>
-

- -

Domain configuration

- -

- In case sanlock loses access to disk locks for some reason, it will - kill all domains that lost their locks. This default behavior may - be changed using - on_lockfailure - element in domain XML. When this element is present, sanlock - will call sanlock_helper (provided by libvirt) with - the specified action. This helper binary will connect to libvirtd - and thus it may need to authenticate if libvirtd was configured to - require that on the read-write UNIX socket. To provide the - appropriate credentials to sanlock_helper, a - client authentication - file needs to contain something like the following: -

-[auth-libvirt-localhost]
-credentials=sanlock
-
-[credentials-sanlock]
-authname=login
-password=password
-

+ The lock manager framework has a pluggable architecture, + to allow different locking technologies to be used. +

+ +

nop: This is a "no op" implementation which does absolutely + nothing. This can be used if mutual exclusion between + virtual machines is not required, or if it is being + solved at another level in the management stack.
lockd: This is the current preferred implementation shipped + with libvirt. It uses the virtlockd daemon + to manage locks using the POSIX fcntl() advisory locking + capability. As such it requires a shared filesystem of + some kind be accessible to all hosts which share the + same image storage.
sanlock: This is an alternative implementation preferred by + the oVirt project. It uses a disk paxos algorithm for + maintaining continuously renewed leases. In the default + setup it requires some shared filesystem, but it is + possible to use it in a manual mode where the management + application creates leases in SAN storage volumes. +

diff --git a/docs/sitemap.html.in b/docs/sitemap.html.in index fad5358c2..5b911ce22 100644 --- a/docs/sitemap.html.in +++ b/docs/sitemap.html.in @@ -101,6 +101,12 @@

Disk locking Ensuring exclusive guest access to disks +

+ Sanlock + Sanlock lock manager plugin +

CGroups -- 2.39.5