Andrea Bolognani [Thu, 25 Jun 2020 12:28:25 +0000 (14:28 +0200)]
ci: Refresh Dockerfiles
All targets get pip3, which is now part of the base system, and
the number of native packages included in the MinGW containers is
significantly reduced.
The corresponding libvirt-ci commit is 4ff697ba0b5d.
Signed-off-by: Andrea Bolognani <abologna@redhat.com>
Peter Krempa [Wed, 24 Jun 2020 09:41:28 +0000 (11:41 +0200)]
kbase: incrementalbackupinternals: Replace bash with pseudocode
Simplify the docs and reduce maintenance burden by just describing the
algorithm by a pseudo-language. Users are encouraged to use libvirt
anyways and projects such as oVirt which do some management of storage
themselves are unlikely to use bash anyways.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Wed, 24 Jun 2020 09:26:29 +0000 (11:26 +0200)]
kbase: incrementalbackupinternals: Add secion on bitmap handling in shell
Add a section that outlines usage of tools to handle bitmaps and
introduce terms corresponding to the output of qemu-img to be used in
further sections.
With this we can simplify the section about checking bitmap health as we
don't have to explain the qemu-img output but can refer to the newly
defined terms.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Laine Stump [Wed, 17 Jun 2020 20:18:23 +0000 (16:18 -0400)]
qemu: auto-assign hostdev devices to PCIe
Until recently, an <interface type='network'> would automatically be
assigned model "rtl8139", which in turn would lead to the device being
assigned a PCI address on a conventional PCI controller (i.e. a
pcie-to-pci-bridge). If the network was a typical Linux host
bridge-based network that used an emulated device, this would be
appropriate, since the guest actually would get an emulated rtl8139
NIC, and that device is a conventional PCI device.
However, if the network being used was a pool of hostdev devices, the
guest would get an actual PCIe network device assigned from the host
via VFIO; while the interface model in that case is irrelevant for the
QEMU commandline to assign the device, the PCI address would have
already been assigned prior to runtime, so the address assignment
would be done based on the model='rtl8139' - a conventional PCI
device. VFIO assignment of a PCIe device to a conventional PCI slot
works, but we would rather have these devices in a PCIe slot.
Since commit bdb8f2e4186, if <interface type='network'> points to a
etwork that is a pool of hostdev devices, the interface model will be
_unset_ by default. This patch uses that information when deciding
what type of slot to assign to the device: since all hostdev network
interfaces are SR-IOV VFs, and *all* SR-IOV network cards are PCIe, it
is safe to assume that the VFs are PCIe and we should assign then to a
PCIe slot in the guest.
Signed-off-by: Laine Stump <laine@redhat.com> Reviewed-by: Andrea Bolognani <abologna@redhat.com>
qemu_domainjob: moved domain job APIs to a separate file
All the domain job related APIs were present in `qemu_domain.c`
along with the other domain APIs. In this patch, we move all the
qemu domain job APIs into a separate file.
Also, in this process, `qemuDomainTrackJob()`,
`qemuDomainFreeJob()`, `qemuDomainInitJob()` and
`qemuDomainObjSaveStatus()` were converted to a non-static
funciton and exposed using `qemu_domain.h`.
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
qemu_domain: Avoid using qemuDomainObjPrivatePtr as parameter
In functions `qemuDomainObjInitJob`, `qemuDomainObjResetJob`,
`qemuDomainObjResetAgentJob`, `qemuDomainObjResetAsyncJob`,
`qemuDomainObjFreeJob`, `qemuDomainJobAllowed`,
`qemuDomainNestedJobAllowed` we avoid sending the complete
qemuDomainObjPrivatePtr as parameter and instead just send
qemuDomainJobObjPtr.
This is done in a effort to separating the qemu-job APIs into
a spearate file.
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Michal Privoznik [Mon, 15 Jun 2020 10:53:48 +0000 (12:53 +0200)]
leaseshelper: Report more errors
Some functions or code paths that may fail don't report error
(e.g. when acquiring PID file fails) leading to a silent quit
of the leaseshelper. This makes it super hard for us and users
to debug what is happening. Fortunately, dnsmasq captures both
stdout and stderr so we can write an error message there.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Michal Privoznik [Thu, 11 Jun 2020 14:43:22 +0000 (16:43 +0200)]
leaseshelper: Wait to acquire PID file
On a DHCP transaction, dnsmasq runs our leases helper which
updates corresponding JSON files. While one dnsmasq won't run the
leaseshelper in parallel, two dnsmasqs (from two distinct
networks) might. To avoid corrupting JSON file, the leaseshelper
acquires PID file first. Well, the way it's acquiring it is not
ideal - it calls virPidFileAcquirePath(wait = false); which
means, that either it acquires the PID file instantly or returns
an error and does not touch the JSON at all. This in turn means
that there might be a leases record missing. With wait = true,
this won't happen.
qemu: fix missing error reports in capabilities probing
The "virsh domcapabilities --arch ppc64" command will fail with no
error message set if qemu-system-ppc64 is not currently installed.
This is because virQEMUCapsCacheLookup() does not report any error
message if not capabilities can be obtained from the cache. Almost
all methods calling this expected an error to be set on failure.
Once that's fixed though, we see a further bug which is that
virQEMUCapsCacheLookupDefault() is passing a NULL binary path to
virQEMUCapsCacheLookup(), so we need to catch that too.
Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Jonathon Jongsma [Tue, 23 Jun 2020 18:29:56 +0000 (13:29 -0500)]
qemu: format 'ramfb' attribute for mediated devices
It's possible to use ramfb as the boot display of an assigned vgpu
device. This was introduced in 4b95738c, but unfortunately the attribute
was not formatted into the xml output for such a device. This patch
fixes that oversight and adds a xml2xml test to verify proper behavior.
The XML format used for QEMU capabilities is not required to be
stable across releases, as we invalidate the cache whenever the
libvirt binary changes.
We none the less always try to parse te entire XML file before
we do any validity checks. Thus if we change the format of any
part of the data, or change permitted values for enums, then
libvirtd logs will be spammed with errors.
These are not in fact errors, but an expected scenario.
This change makes the loading code validate the cache timestamp
against the libvirtd timestamp immediately. If they don't match
then we stop loading the rest of the XML file.
Reviewed-by: Michal Privoznik <mprivozn@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
It is easier for management software (and subsequently
distributions) to install hook script under
/etc/libvirt/hooks/$driver.d/ and have libvirt execute them in
alphabetical order. To maintain backwards compatibility,
/etc/libvirt/hooks/$driver hook script is executed the first
followed by scripts from the $driver.d directory.
The stdio is chained between the scripts. The output of the first
script is input of the second and so on.
Signed-off-by: Dmitry Nesterenko <dmitry.nesterenko@virtuozzo.com> Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
This patch wraps it up all the wiring done in previous patches,
enabling a PPC64 guest to launch a guest using a TPM Proxy
device.
Note that device validation is already being done in qemu_validate.c,
qemuValidateDomainDeviceDefTPM(), on domain define time. We don't
need to verify QEMU capabilities for this device again inside
qemu_command.c.
Tested-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com> Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> Signed-off-by: Ján Tomko <jtomko@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
tests: add XML schema tests for the TPM Proxy device
This tests aims to exercise how a TPM Proxy device can be
added in the domain, either alone or with a regular TPM
device. It also ensures that we do not allow bogus scenarios
to slip by.
Tested-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com> Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Ján Tomko <jtomko@redhat.com>
qemu: add validations after TPM Proxy model introduction
Previous patch handled the conversion of def->tpm to the
array def->tpms and the XML parsing logic. This patch handles
the validations needed to ensure the intended behavior.
The existing qemuValidateDomainDeviceDefTPM() function was updated
to guarantee that the VIR_DOMAIN_TPM_MODEL_SPAPR_PROXY model is
exclusive to PPC64 guests and to the VIR_DOMAIN_TPM_TYPE_PASSTHROUGH
backend.
A new function called qemuDomainDefTPMsPostParse() was added to guarantee
that the following combinations in the same domain are valid:
- a single TPM device
- a single TPM Proxy device
- a single TPM + single TPM Proxy devices
And these combinations in the same domain are NOT valid:
- 2 or more TPM devices
- 2 or more TPM Proxy devices
Tested-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com> Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Ján Tomko <jtomko@redhat.com>
A TPM Proxy device can coexist with a regular TPM, but the
current domain definition supports only a single TPM device
in the 'tpm' pointer. This patch replaces this existing pointer
in the domain definition to an array of TPM devices.
All files that references the old pointer were adapted to
handle the new array instead. virDomainDefParseXML() TPM related
code was adapted to handle the parsing of an extra TPM device.
TPM validations after this new scenario will be updated in
the next patch.
Tested-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com> Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> Signed-off-by: Ján Tomko <jtomko@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
qemu: Extend QEMU capabilities with 'spapr-tpm-proxy'
Expose the TPM Proxy support for PPC64 guests by creating a new
cap called QEMU_CAPS_DEVICE_SPAPR_TPM_PROXY.
This device is part of the machinery the guest need to orchestrate
with the PPC64 Ultravisor the transition to the Secure VM (SVM)
mode. Inside QEMU, this device will be used with the H_TPM_COMM
hypercall to connect with the TPM Resource Manager, enabling
the guest to open and close TPM sessions with the host TPM.
Tested-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com> Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel Henrique Barboza <danielhb413@gmail.com> Signed-off-by: Ján Tomko <jtomko@redhat.com>
docs: documentation and schema for the new TPM Proxy model
QEMU 4.1.0 introduced a new device type called TPM Proxy, currently
implemented by PPC64 guests via a new virtual device called
'spapr-tpm-proxy' (see QEMU 0fb6bd073230 for more info).
The TPM Proxy device interacts with a TPM Resource Manager, a host
device capable of multiplexing the host TPM with multiple processes.
This allows multiple guests to access some TPM features at the
same time. Note that this mode of operation does not provide
full TPM features to be available for the guest - for that case
the guest still needs to assign a vTPM device (tpm-spapr for
PPC64 guests). Although redundant, there is currently no technical
limitation for a guest to assign both a vTPM and a TPM Proxy at the
same time.
This patch adds documentation and schema for a new TPM model
type called 'spapr-tpm-proxy' that creates this new TPM Proxy
device. This model is valid only for the 'passthrough' backend.
An example of a TPM Proxy device connected to a TPM Resource Manager
'/dev/tpmrm0' will look like this:
Michal Privoznik [Tue, 23 Jun 2020 13:55:42 +0000 (15:55 +0200)]
qemu_command: Fix IOMMU validator name in two comments
When building command line for IOMMU or machine, there are two
comments which mention function that validate IOMMU. But they
both refer to old name which was changed in v6.3.0-rc1~246.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Integrate both 'disk-hostdev-scsi-virtio-iscsi-auth-AES' and
'hostdev-scsi-virtio-iscsi-auth' as the new test infrastructure tests
both legacy and 'secret' object cases.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Peter Krempa [Wed, 17 Jun 2020 13:24:05 +0000 (15:24 +0200)]
qemuxml2argvtest: hostdev-scsi-lsi: Integrate 'hostdev-scsi-lsi-iscsi-auth' case
We can add the authenticated iSCSI hostdevs to the same test file.
Additionally this now covers passing secret via the 'secret' object
rather than on the command line.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Andrea Bolognani [Tue, 23 Jun 2020 09:22:49 +0000 (11:22 +0200)]
ci: Drop mips container for Debian sid
The build job for this container has been failing every single
time, and as it turns out the explanation for that is very simple:
Debian is just not going to support the mips architecture going
forward.
Reported-by: Pino Toscano <ptoscano@redhat.com> Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Peter Krempa [Tue, 23 Jun 2020 05:33:41 +0000 (07:33 +0200)]
qemuBackupBegin: Don't leak 'def' on early failures
The cleanup path expects that 'def' is assigned to 'priv->backup', but
that's not the case for early failures. Add a check to stop overwriting
of 'def' so that it can be freed.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
The above message looks like we've forgot to print the error message
itself as the line ends after 'error:'. Unfortunately with the current
API design clients have no way of actually getting the error message as
the VIR_DOMAIN_EVENT_ID_BLOCK_JOB(_2) event only reports the status but
not an error and the job then vanishes.
Fix the expectations by using vshPrintExtra instead of vshError:
Peter Krempa [Fri, 29 May 2020 11:03:53 +0000 (13:03 +0200)]
qemublocktest: Add 'snapshots' tests for block commit bitmap handling
Simulate commit between all the combinations of layers in the
'snapshots' case to see whether the code merges the correct bitmaps with
the correct depth of temporary bitmaps.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Fri, 17 Apr 2020 07:51:40 +0000 (09:51 +0200)]
qemu: Rewrite bitmap handling for block commit
Reuse qemuBlockGetBitmapMergeActions which allows removing the ad-hoc
implementation of bitmap merging for block commit. The new approach is
way simpler and more robust and also allows us to get rid of the
disabling of bitmaps done prior to the start as we actually do want to
update the bitmaps in the base.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Fri, 29 May 2020 10:55:02 +0000 (12:55 +0200)]
qemublocktest: Add 'snapshots' tests for backup bitmap handling
The 'snapshots' case has multiple layers so we need to make sure that
the bitmaps are merged with the appropriate temporary bitmaps formatted
from the allocation bitmap for any backing chain layer above.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Fri, 22 May 2020 12:48:46 +0000 (14:48 +0200)]
qemu: backup: Rewrite backup bitmap handling to the new bitmap semantics
Reuse qemuBlockGetBitmapMergeActions which allows removal of the ad-hoc
implementation of bitmap merging for backup. The new approach is simpler
and also more robust in case some of the bitmaps break as they remove
the dependency on the whole chain of bitmaps working.
The new approach also allows backups if a snapshot is created outside of
libvirt.
Additionally the code is greatly simplified.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Mon, 1 Jun 2020 10:05:49 +0000 (12:05 +0200)]
qemu: block: Add universal helper for merging dirty bitmaps for all scenarios
Add a function which allows merging bitmaps according to the new
semantics and will allow replacing all the specific ad-hoc functions
currently in use for 'backup', 'block commit', 'block copy' and will
also be usable in the future for 'block pull' and non-shared storage
migration.
The semantics are a bit quirky for the 'backup' case but these quirks
are documented and will prevent us from having two slightly different
algorithms.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Tue, 21 Apr 2020 06:17:58 +0000 (08:17 +0200)]
qemu: checkpoint: Don't merge checkpoints during deletion
Now that we've switched to the simple handling, the first thing that can
be massively simplified is checkpoint deletion. We now need to only go
through the backing chain and find the appropriately named bitmaps and
delete them, no complex lookups or merging.
Note that compared to other functions this deletes the bitmap in all
layers compared to others where we expect only exactly 1 bitmap of a
name in the backing chain to prevent potential problems.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Tue, 2 Jun 2020 13:11:25 +0000 (15:11 +0200)]
qemublocktest: Add new 'synthetic' bitmap detection and validation test case
Based on the 'snapshots' example with manual tweaks to introduce
inactive, transient, inconsistent and duplicate bitmaps in various parts
of the chain to exercise detection and new validation code.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Fri, 17 Apr 2020 07:18:49 +0000 (09:18 +0200)]
qemu: checkpoint: Don't chain bitmaps for checkpoints
Chaining bitmaps for checkpoints (disabling the active one and creating
a new) severely overcomplicated all operations in regards to bitmaps.
Specifically it requires us re-matching the on-disk state to the
internal metadata and in case of merging during block jobs it makes it
almost impossible to cover all corner cases.
Since the checkpoints and incremental backups were not yet enabled,
let's change the design to keep one bitmap per checkpoint. In case of
layered snapshots this will be filled in by using dirty-bitmap-populate.
Finally the main reason for this unnecessary complexity was the fear
that qemu's performance could degrade. In the end I think that
addressing the performance issue will be better done in qemu (e.g by
keeping an internal bitmap updated with changes and merging it
periodically back to the real bitmaps. QEMU writes out changes to disk
at shutdown so consistency is not a problem).
Removing the relationships between bitmaps frees us from complex
handling and also makes all the surrounding code more robust as one
broken bitmap doesn't necessarily invalidate whole chains of backups.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Tue, 2 Jun 2020 07:11:36 +0000 (09:11 +0200)]
qemublocktest: Extract printing of nodename list
There will be multiple places where we'll need to print nodenames from a
GSList of virStorageSource for testing purposes. Extract the code into a
function.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Thu, 28 May 2020 15:02:18 +0000 (17:02 +0200)]
qemublocktest: Disable testcases for all bitmap handling
Upcoming patches are going to rewrite and semantically modify how
bitmaps are handled during blockjobs. This is possible as incremental
backup is not yet fully enabled.
As the changes are going to be incompatible with any current test data
remove all test cases for bitmap handling during checkpoint deletion,
incremental backups, block commit, block copy, and bitmap validation
operations.
The tests will be gradually added back later after the code and
test-data is refactored.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Thu, 28 May 2020 14:25:55 +0000 (16:25 +0200)]
qemublocktest: Add 'empty' case for incremental backup test
Use the new test data when calculating incremental backup operations. As
incremental backup fails with no bitmap the test code is modified to
allow testing this case too.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Thu, 21 May 2020 11:23:40 +0000 (13:23 +0200)]
qemu: backup: Fix backup of disk skipped in an intermediate checkpoint
If a disk is not captured by one of the intermediate checkpoints the
code would fail, but we can easily calculate the bitmaps to merge
correctly by skipping over checkpoints which don't describe the disk.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Peter Krempa [Tue, 28 Apr 2020 13:03:11 +0000 (15:03 +0200)]
qemu: backup: Split up code traversing checkpoint list looking for bitmaps
The algorithm is getting quite complex. Split out the lookup of range of
backing chain storage sources and bitmaps contained in them which
correspond to one checkpoint.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com>
Jiri Denemark [Wed, 17 Jun 2020 12:42:59 +0000 (14:42 +0200)]
cpu_map: Request test files update when adding x86 features
The CPUID data in *-{disabled,enabled}.xml convert feature names from
the corresponding *.json file into raw CPUID and MSR data and thus some
of them may need to be updated when new features are added into the CPU
map.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Erik Skultety [Fri, 19 Jun 2020 12:43:24 +0000 (14:43 +0200)]
nodedev: Makefile: Use AM_LDFLAGS_MOD_NOUNDEF with the installed library
Rather than using the AM_LDFLAGS_MOD_NOUNDEF options with the noinstall
library that will come out of libtool from
libvirt_driver_nodedev_impl_la, use it with the installed version
libvirt_driver_nodedev_la.
Broken-by-commit: c44bffb9 Fixes: https://ci.centos.org/job/libvirt-rpm/systems=libvirt-fedora-32/1155/ Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Jonathon Jongsma [Thu, 18 Jun 2020 21:06:04 +0000 (16:06 -0500)]
docs: note node device fields that are read-only
As noted by Erik Skultety, we use the same XML schema to report
existing devices and to define new devices. However, some schema
elements are "read-only". In other words, they are used to report
information from the node device driver and cannot be used to define a
new device. Note these in the documentation.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
projects / libvirt.git / log