Ján Tomko [Wed, 20 Nov 2019 18:13:29 +0000 (19:13 +0100)]
scripts: check-aclrules: use regular expressions less often
Use a simple
if "substr" in line
before running a regular expression, which saves time,
especially if the regex has a capture group.
This reduces runtime of the check by almost 78 % for me.
Signed-off-by: Ján Tomko <jtomko@redhat.com> Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com> Reviewed-by: Erik Skultety <eskultet@redhat.com>
Ján Tomko [Wed, 20 Nov 2019 18:05:13 +0000 (19:05 +0100)]
scripts: check-aclrules: use in instead of find
For checking whether a substring is present in a string,
using the pattern:
"str" in string
is slightly faster than:
string.find("str") != -1
Use it to shave off 4 % of the runtime of this script that
processes quite a few long source files.
Signed-off-by: Ján Tomko <jtomko@redhat.com> Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com> Reviewed-by: Erik Skultety <eskultet@redhat.com>
Ján Tomko [Wed, 9 Oct 2019 19:02:38 +0000 (21:02 +0200)]
scripts: speedup prohibit-duplicate-header
Running regular expressions with capture groups is expensive.
Bail out early if the line does not start with a '#'.
This reduces the runtime of the check by two thirds.
Signed-off-by: Ján Tomko <jtomko@redhat.com> Reviewed-by: Daniel Henrique Barboza <danielhb413@gmail.com> Reviewed-by: Erik Skultety <eskultet@redhat.com>
Michal Privoznik [Mon, 18 Nov 2019 16:40:01 +0000 (17:40 +0100)]
qemuProcessStop: Remove image metadata for running mirror jobs
If user starts a blockcommit or a blockcopy then we modify access
for qemu on both images and leave it like that until the job
terminates. So far so good. Problem is, if user instead of
terminating the job (where we would modify the access again so
that the state before the job is restored) calls destroy on the
domain or if qemu dies whilst executing the block job. In this
case we don't ever clear the access we granted at the beginning.
To fix this, maybe a bit harsh approach is used, but it works:
after all labels were restored (that is after
qemuSecurityRestoreAllLabel() was called), we iterate over each
disk in the domain and remove XATTRs from the whole backing chain
and also from any file the disk is being mirrored to.
This would have been done at the time of pivot, but it isn't
because user decided to kill the domain instead. If we don't do
this and leave some XATTRs behind the domain might be unable to
start.
Also, secdriver can't do this because it doesn't know if there is
any job running. It's outside of its scope - the hypervisor
driver is responsible for calling secdriver's APIs.
Moreover, this is safe to call because we don't remember labels
for any member of a backing chain except of the top layer. But
that one was restored in qemuSecurityRestoreAllLabel() call done
earlier. Therefore, not only we don't remember labels (and thus
this is basically a NOP for other images in the backing chain) it
is also safe to call this when no blockjob was started in the
first place, or if some parts of the backing chain are shared
with some other domains - this is NOP, unless a block job is
active at the time of domain destroy.
Michal Privoznik [Tue, 19 Nov 2019 07:43:58 +0000 (08:43 +0100)]
qemu: Separate image metadata removal into a function
There are four places where we remove image XATTRs and in all of
them we have the same for() loop with the same body. Move it into
a separate function because I'm about to introduce fifth place
where the same needs to be done.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Peter Krempa [Fri, 15 Nov 2019 15:25:26 +0000 (16:25 +0100)]
qemu: domain: Introduce helper to convert <loader> into virStorageSource
Add a helper which will covert the PFLASH code file and variable file
into the virStorageSource objects stored in private data so that we can
use them with -blockdev while keeping the infrastructure to determine
the path to the loaders intact.
This is a temporary solution until we will want to do snapshots of the
pflash where we will be forced do track the full backing chain in the
XML.
In the meanwhile just convert it partially so that we can stop using
-drive.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Peter Krempa [Fri, 15 Nov 2019 15:23:44 +0000 (16:23 +0100)]
qemu: domain: Store virStorageSources representing pflash backing
To allow converting the pflash drives to blockdev we will need a
virStorageSource to allow using our helpers. Temporarily prior to
coverting loader data to a virStorageSoruce add private data which will
house this.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Peter Krempa [Fri, 1 Nov 2019 11:14:43 +0000 (12:14 +0100)]
qemu: command: Extract formatting of -drive for pflash
Extract the old way to instantiate pflash devices to hold the firmware
via -drive to a separate function so that it can later be conditionally
disabled when -blockdev will be used.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Thu, 21 Nov 2019 08:52:24 +0000 (09:52 +0100)]
domcaps: Remove function initializing domain caps as unsupported
Commit 5751a0b6b1968bb2354b2ac21cc5938b93009590 added a helper function
called virDomainCapsFeaturesInitUnsupported which initialized all domain
capability features as unsupported.
When adding a new feature this would initialize it as unsupported also
for hypervisor drivers which the original author possibly didn't intend
to modify. To prevent accidental wrong value being reported in such case
revert back to initializing individual features in the hypervisor
drivers themselves.
This is not a straight revert as additonal patches modified how we store
the capabilities.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Erik Skultety [Mon, 18 Nov 2019 11:04:16 +0000 (12:04 +0100)]
libvirt-<module>: Check caller-provided buffers to be NULL with size > 0
Pre-Glib era which used malloc allowed the size of the client-side
buffers to be declared as 0, because malloc documents that it can either
return 0 or a unique pointer on 0 size allocations.
With glib this doesn't work anymore, because glib documents that for
such allocation requests NULL is always returned which results in an
error in our public API checks server-side.
This patch complements the fix in the RPC layer by explicitly erroring
out on the following combination of args used by our legacy APIs (their
moder equivalents don't suffer from this):
Ján Tomko [Mon, 11 Nov 2019 09:21:03 +0000 (10:21 +0100)]
conf: remove NULL check from virDomainQemuMonitorEventNew
The qemu_domain_monitor_event_msg struct in qemu_protocol.x
defines event as a nonnull_string and qemuMonitorJSONIOProcessEvent
also errors out on a non-NULL event.
Drop the check to fix the build with static analysis.
Shared memory devices need qemu to be able to access certain paths
either for the shared memory directly (mostly ivshmem-plain) or for a
socket (mostly ivshmem-doorbell).
Add logic to virt-aa-helper to render those apparmor rules based
on the domain configuration.
apparmor: let AppArmorSetSecurityImageLabel append rules
There are currently broken use cases, e.g. snapshotting more than one disk at
once like:
$ virsh snapshot-create-as --domain eoan --disk-only --atomic
--diskspec vda,snapshot=no --diskspec vdb,snapshot=no
--diskspec vdc,file=/test/disk1.snapshot1.qcow,snapshot=external
--diskspec vdd,file=/test/disk2.snapshot1.qcow,snapshot=external
The command above will iterate from qemuDomainSnapshotCreateDiskActive and
eventually add /test/disk1.snapshot1.qcow first (appears in the rules)
to then later add /test/disk2.snapshot1.qcow and while doing so throwing
away the former rule causing it to fail.
All other calls to (re)load_profile already use append=true when adding
rules append=false is only used when restoring rules [1].
Fix this by letting AppArmorSetSecurityImageLabel use append=true as well.
Since this is removing a (unintentional) trigger to revoke all rules
appended so far we agreed on review to do some tests, but in the tests
no rules came back on:
- hot-plug
- hot-unplug
- snapshotting
A lot of the code in AppArmorSetSecurityImageLabel is a duplicate of
what is in reload_profile, this refactors AppArmorSetSecurityImageLabel
to use reload_profile instead.
reload_profile calls get_profile_name for no particular gain, lets
remove that call. The string isn't used in that function later on
and not registered/passed anywhere.
It can only fail if it either can't allocate or if the
virDomainDefPtr would have no uuid set (which isn't allowed).
Thereby the only "check" it really provides is if it can allocate the
string to then free it again.
This was initially added in [1] when the code was still in
AppArmorRestoreSecurityImageLabel (later moved) and even back then had
no further effect than described above.
While only used internally from libvirt the options still are misleading
enough to cause issues every now and then.
Group modes, options and an adding extra file and extend the wording of
the latter which had the biggest lack of clarity.
Both add a file to the end of the rules, but one re-generates the
rules from XML and the other keeps the existing rules as-is not
considering the XML content.
When starting a domain without a CPU model specified in the domain XML,
QEMU will choose a default one. Which is fine unless the domain gets
migrated to another host because libvirt doesn't perform any CPU ABI
checks and the virtual CPU provided by QEMU on the destination host can
differ from the one on the source host.
With QEMU 4.2.0 we can probe for the default CPU model used by QEMU for
a particular machine type and store it in the domain XML. This way the
chosen CPU model is more visible to users and libvirt will make sure
the guest will see the exact same CPU after migration.
Architecture specific notes
- aarch64: We only set the default CPU for TCG domains as KVM requires
explicit "-cpu host" to work.
- ppc64: The default CPU for KVM is "host" thanks to some hacks in QEMU,
we will translate the default model to the model corresponding to the
host CPU ("POWER8" on a Power8 host, "POWER9" on Power9 host, etc.).
This is not a problem as the corresponding CPU model is in fact an
alias for "host". This is probably not ideal, but it's not wrong and
the default virtual CPU configured by libvirt is the same QEMU would
use. TCG uses various CPU models depending on machine type and its
version.
- s390x: The default CPU for KVM is "host" while TCG defaults to "qemu".
- x86_64: The default CPU model (qemu64) is not runnable on any host
with KVM, but QEMU just disables unavailable features and starts
happily.
Jiri Denemark [Thu, 24 Oct 2019 12:19:38 +0000 (14:19 +0200)]
qemu: Make probed machine types depend on accelerator
Some specifics of machine types may depend on the accelerator and thus
the data should be moved to virQEMUCapsAccel. The TCG machine types are
just copied from the ones probed for KVM to simplify the changes to
qemucapabilitiestest data files.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Jiri Denemark [Thu, 24 Oct 2019 07:04:58 +0000 (09:04 +0200)]
qemu: Move machine type data in capabilities cache
In preparation for making machine types dependent on the accelerator,
the <machine> elements are formatted between <cpu type='kvm'> and
<cpu type='tcg'>.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
It is a tiny wrapper around virQEMUCapsProbeQMPCPUDefinitions which will
soon get private parameters and thus it cannot be exposed outside
qemu_capabilities.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
We will need to keep some QEMU-specific data for each CPU model
supported by a QEMU binary. Instead of complicating the generic
virDomainCapsCPUModelsPtr, we can just directly store
qemuMonitorCPUDefsPtr returned by the capabilities probing code.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Jiri Denemark [Wed, 9 Oct 2019 08:16:17 +0000 (10:16 +0200)]
qemu: Split virQEMUCapsFetchCPUModels
Most of the code moved to a new virQEMUCapsFetchCPUDefinitions function
and the existing virQEMUCapsFetchCPUModels just becomes a small wrapper
around virQEMUCapsFetchCPUDefinitions and virQEMUCapsCPUDefsToModels.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Jiri Denemark [Wed, 9 Oct 2019 08:14:59 +0000 (10:14 +0200)]
qemu: Rename virQEMUCaps{Get,Fetch}CPUDefinitions
The functions return virDomainCapsCPUModelsPtr and thus they should be
called *CPUModels for consistency. Functions called *CPUDefinitions will
work on qemuMonitorCPUDefsPtr.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Jiri Denemark [Tue, 29 Oct 2019 13:36:11 +0000 (14:36 +0100)]
qemu: Use virDomainCapsCPUUsable in qemuMonitorCPUDefInfo
While virDomainCapsCPUModel structure contains 'usable' field with
virDomainCapsCPUUsable type, the lower level structure specific to QEMU
driver used virTriStateBool for the same thing and we had to translate
between them.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Jiri Denemark [Thu, 10 Oct 2019 14:44:40 +0000 (16:44 +0200)]
qemu: Change return type of virQEMUCapsFetchCPUDefinitions
The function would return a valid virDomainCapsCPUModelsPtr with empty
CPU models list if query-cpu-definitions exists in QEMU, but returns
GenericError meaning it's not in fact implemented. This behaviour is a
bit strange especially after such virDomainCapsCPUModels structure is
stored in capabilities XML and parsed back, which will result in NULL
virDomainCapsCPUModelsPtr rather than a structure containing nothing.
Let's just keep virDomainCapsCPUModelsPtr NULL if the QMP command is not
implemented and change the return value to int so that callers can
easily check for failure or success.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
qemu: Filter models in virQEMUCapsGetCPUDefinitions
Some callers of virQEMUCapsGetCPUDefinitions will need to filter the
returned list of CPU models. Let's add the filtering parameters directly
to virQEMUCapsGetCPUDefinitions to avoid copying the CPU models list
twice.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
qemu: Copy CPU models in virQEMUCapsGetCPUDefinitions
Rather than returning a direct pointer the list stored in qemuCaps the
function now creates a new copy of the CPU models list.
The main purpose of this seemingly useless change is to update callers
to free the result returned by virQEMUCapsGetCPUDefinitions because the
internals of this function will change significantly in the following
patches.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the genpolkit.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the check-aclrules.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the check-driverimpls.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the check-drivername.pl tool in Python.
This was mostly a straight conversion, manually going line-by-line
to change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
In testing though it was discovered the existing code was broken
since it hadn't been updated after driver.h was split into many
files. Since the old code is being thrown away, the fix was done
as part of the rewrite rather than split into a separate commit.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
src: rewrite systemtap function generator in Python
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the gensystemtap.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the dtrace2systemtap.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
The "--with-modules" flag was dropped because this functionality
is not implicitly always enabled.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the check-symfile.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the check-symsorting.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the check-aclperms.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the header-ifdef.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
As part of a goal to eliminate Perl from libvirt build tools,
rewrite the mock-noinline.pl tool in Python.
This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.
Tested-by: Cole Robinson <crobinso@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Another weird bug appeared concerning qemu namespaces. Basically
the problem is as follows:
1) Issue an API that causes libvirt to create a node in domain's
namespace, say /dev/nvme0n1 with 8:0 as major:minor (the API can
be attach-disk for instance). Or simply create the node from a
console by hand.
2) Detach the disk from qemu.
3) Do something that makes /dev/nvme0n1 change it's minor number.
4) Try to attach the disk again.
The problem is, in a few cases - like disk-detach - we don't
remove the corresponding /dev node from the mount namespace
(because it may be used by some other disk's backing chain). But
this creates a problem, because if the node changes its MAJ:MIN
numbers we don't propagate the change into the domain's
namespace. We do plain mknod() and ignore EEXIST which obviously
is not enough because it doesn't guarantee that the node has
updated MAJ:MIN pair.
Peter Krempa [Thu, 26 Sep 2019 14:56:16 +0000 (16:56 +0200)]
qemu: checkpoint: Fix rollback and access to unlocked 'vm' when deleting checkpoints
Delete/merge bitmaps when deleting checkpoints using a 'transaction' so
that we don't have to deal with halfway-failed scenarios and also fix
access to 'vm' while in the monitor lock.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Peter Krempa [Tue, 19 Nov 2019 10:55:05 +0000 (11:55 +0100)]
tests: virschema: Propagate errors from directory traversal in testSchemaDir
testSchemaDir is a helper which invokes the schema test using virTestRun
on all schema files. Since the function itself is not called inside
virTestRun any helper function call is not dispatched to the user and
thus it's hard to debug the test. Propagate errors from the directory
traversal.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Peter Krempa [Tue, 19 Nov 2019 10:52:40 +0000 (11:52 +0100)]
tests: utils: Introduce helper for dispatching libvirt errors
In cases when we call a libvirt helper which reports an error the error
would be hidden unless libvirt library debug is on. This produces a lot
of output and is hard to debug.
The helper provides a way to dispatch the libvirt error in specific
cases sice we do already dispatch it in case when virTestRun is used.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
projects / libvirt.git / log