Andrea Bolognani [Tue, 19 Sep 2023 17:28:55 +0000 (19:28 +0200)]
systemd: Introduce common templates
We already use templating to generate sockets, which are all
based off libvirtd's. Push the idea further, and extend it to
cover services as well.
This is more challenging, as the various modular daemons each have
their own needs in terms of what system services needs to be
available before they can be started, which other components of
libvirt they depend on, and so on.
In order to make this sort of per-service tweaks possible, we
introduce a Python script that can merge two systemd units
together. The script is aware of the semantics of systemd's unit
definition format, so it can intelligently merge sections
together.
This generic systemd unit merging mechanism will also supersede
the extremely ad-hoc @deps@ variable, which is currently used in
a single scenario.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Andrea Bolognani [Thu, 21 Sep 2023 15:03:19 +0000 (17:03 +0200)]
systemd: Provide all input files explicitly
We're about to change the defaults and start migrating to common
templates: in order to be able to switch units over one at a
time, make the input files that are currently used explicit
rather than implicit.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
These will be useful during the upcoming migration to common
templates for systemd units and will be dropped as soon as all
services have been converted.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Andrea Bolognani [Tue, 19 Sep 2023 19:07:28 +0000 (21:07 +0200)]
systemd: Drop Conflicts from virtproxyd sockets
The idea behind these is to prevent running both modular daemons
and monolithic daemon at the same time. We will implement a more
effective solution for that shortly.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Erik Skultety [Tue, 19 Sep 2023 09:26:01 +0000 (11:26 +0200)]
gitpublish: Add suppresscc option
send-email scans the commit messages to figure out the default set of
addresses to put into CC, Acked-by/Reviewed-by, etc-by being among
them. We're quite strict about CC-ing people on libvirt-list, since
most developers are subscribed to the list anyway. Respect the rule by
avoiding CCing people solely based on the fact that they've done review
of any of previous revisions.
Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Andrea Bolognani [Wed, 27 Sep 2023 17:39:47 +0000 (19:39 +0200)]
docs: Go bindings release at the same time as the C library
The actual versioning policy[1] is a bit more nuanced, and in
particular there are scenarios in which the monthly release
is intentionally skipped, but overall it's not inaccurate to
claim that the release cadence of the Go bindings follows the
one of the C library.
So we still need to be able to provide a custom @sockprefix@ in
those cases, but in the most common scenario we can do away with
the requirement by introducing a sensible default.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
The meaning of the _def suffix might not be immediately obvious,
especially since it's also used to refer to the output of the
meson-gen-def.py script elsewhere in the same file.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Andrea Bolognani [Tue, 19 Sep 2023 14:24:44 +0000 (16:24 +0200)]
systemd: Set @name@ for virtlogd/virtlockd
The information is not used anywhere right now, but the
documentation for virt_daemon_units claims it's mandatory.
We also intend to actually start using it later on.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Andrea Bolognani [Mon, 18 Sep 2023 13:25:28 +0000 (15:25 +0200)]
systemd: Add missing WantedBy for virtlogd/virtlockd
This annotation being missing resulted in virtlogd and virtlockd
being marked as "indirect" services, i.e. services that cannot
be started directly but have to be socket activated instead.
While this is our preferred configuration, we shouldn't prevent
the admin to start them at boot if they want to.
Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
util: Fix error return for virProcessKillPainfullyDelay()
Commit 93af79fb removed a cleanup label in favor of returning error
values directly in certain cases. But the final return value was changed
from -1 to 0. If we get to the end of the function, that means that
we've waited for the process to exit but it still exists. So we should
return -1. The error message was still being set correctly, but we were
returning a success status (0).
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Jim Fehlig [Fri, 22 Sep 2023 18:20:35 +0000 (12:20 -0600)]
libxl: Fix connection to modular network daemon
In a modular daemon configuration, virtxend does not support the
virNetwork* APIs. It should open a connection to virtnetworkd when
using those APIs, but currently always opens a connection to
"xen:///system". Switch to using virGetConnectNetwork to obtain a
valid connection instead of using the hardcoded URI.
Signed-off-by: Jim Fehlig <jfehlig@suse.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
interface: fix udev_device_get_sysattr_value return value check
Reviewing the code I found that return value of function
udev_device_get_sysattr_value() is dereferenced without a check.
udev_device_get_sysattr_value() may return NULL by number of reasons.
v2: VIR_DEBUG added, replaced STREQ(NULLSTR()) with STREQ_NULLABLE()
v3: More checks added, to skip earlier. More verbose VIR_DEBUG.
Signed-off-by: Dmitry Frolov <frolov@swemel.ru> Reviewed-by: Martin Kletzander <mkletzan@redhat.com>
Michal Privoznik [Fri, 22 Sep 2023 09:01:40 +0000 (11:01 +0200)]
virDomainMemoryDefValidate: Check for overlapping memory devices
As of v9.4.0-rc2~5 it is possible to specify guest address where
a virtio-mem/virtio-pmem memory device is mapped to. What that
commit forgot to introduce was a check for overlaps.
And yes, this is technically an O(n^2) algorithm, as
virDomainMemoryDefValidate() is called over each memory device
and after this, virDomainMemoryDefValidate() also iterates over
each memory device. But given there's usually only a handful of
such devices, and this runs only when parsing domain XML I guess
code readability wins over some less obvious solution.
Resolves: https://issues.redhat.com/browse/RHEL-4452 Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Inside of virDomainMemoryDefValidate() there's a check that
address where a virtio-mem memory device is mapped to is a
multiple of its block size. But this check is off by a couple of
bits, because the memory address is in bytes while the block size
is in kibibytes. Therefore, when checking whether address is a
multiple of the block size, the latter has to be multiplied by a
factor of 1024.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
QEMU mandates the VIRTIO_PMEM address is aligned to a pagesize.
This is a very reasonable requirement. So much so, that it
deserves to be in hypervisor agnostic validation code
(virDomainMemoryDefValidate()). Not that any other hypervisor
would support VIRTIO_PMEM yet. But even if they did, this would
surely be still valid.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Andrea Bolognani [Fri, 22 Sep 2023 13:19:46 +0000 (15:19 +0200)]
qemu: Improve error message for failed firmware autoselection
The current message can be misleading, because it seems to suggest
that no firmware of the requested type is available on the system.
What actually happens most of the time, however, is that despite
having multiple firmwares of the right type to choose from, none
of them is suitable because of lacking some specific feature or
being incompatible with some setting that the user has explicitly
enabled.
Providing an error message that describes exactly the problem is
not feasible, since we would have to list each candidate along
with the reason why we rejected it, which would get out of hand
quickly.
As a small but hopefully helpful improvement over the current
situation, reword the error message to make it clearer that the
culprit is not necessarily the firmware type, but rather the
overall domain configuration.
Suggested-by: Michael Kjörling <7d1340278307@ewoof.net> Signed-off-by: Andrea Bolognani <abologna@redhat.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Our test driver lacks implementation for
virConnectGetDomainCapabilities(). Provide one, though a trivial
one. Mostly so that something else than VIR_ERR_NO_SUPPORT error
is returned.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Michal Privoznik [Mon, 18 Sep 2023 14:15:33 +0000 (16:15 +0200)]
virerror: Make virReportEnumRangeError() check for type mismatch
As can be seen from previous commits, it's fairly easy to pass a
different type to virReportEnumRangeError() than the actual
variable is of. So far, we have a sizeof() hack to check if some
nonsensical types are not passed, e.g. it catches cases where a
function name is passed instead of an enum. Extend the hack to
check whether proper enum was passed.
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Michal Privoznik [Mon, 18 Sep 2023 13:45:52 +0000 (15:45 +0200)]
virNetDevVPortProfileOp8021Qbh: Use proper type in virReportEnumRangeError()
The @virtPortOp variable inside of virNetDevVPortProfileOp8021Qbh
is of type virNetDevVPortProfileLinkOp. Pass the proper type to
virReportEnumRangeError().
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Michal Privoznik [Mon, 18 Sep 2023 13:45:12 +0000 (15:45 +0200)]
virnetdevvportprofile: Turn virNetDevVPortProfileLinkOp enum into a proper typedef
This allows us to declare variables without using 'enum
virNetDev....' and will become more useful in the near future
(when virReportEnumRangeError() is fixed).
Signed-off-by: Michal Privoznik <mprivozn@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
qemu: implement ssh-agent auth for ssh disks with nbdkit
It's not possible to use password-protected ssh keys directly with
libvirt because libvirt doesn't have any way to prompt a user for the
password. To accomodate password-protected key files, an administrator
can add these keys to an ssh agent and then configure the domain with
the path to the ssh-agent socket.
Note that this requires an administrator or management app to
configure the ssh-agent with an appropriate socket path and add the
necessary keys to it. In addition, it does not currently work with
selinux enabled. The ssh-agent socket would need a label that libvirt
would be allowed to access rather than unconfined_t.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jonathon Jongsma [Thu, 22 Dec 2022 22:56:47 +0000 (16:56 -0600)]
qemu: implement keyfile auth for ssh disks with nbdkit
For ssh disks that are served by nbdkit, we can support logging in with
an ssh key file. Pass the path to the configured key file and the
username to the nbdkit process.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jonathon Jongsma [Thu, 19 Jan 2023 21:52:20 +0000 (15:52 -0600)]
schema: add keyfile configuration for ssh disks
Authenticating via key file to an ssh server is often preferable to
logging in via password. In order to support this functionality add a
new <identity> xml element for ssh disks that allows the user to specify
a keyfile and username. Example configuration:
Jonathon Jongsma [Thu, 19 Jan 2023 21:46:22 +0000 (15:46 -0600)]
schema: add configuration for host verification of ssh disks
In order to make ssh disks usable, we need to be able to validate a
remote host. To do this, add a <knownHosts> xml element for ssh disks to
allow the user to specify a location for a file that contains known host
keys. Implementation to follow.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jonathon Jongsma [Wed, 21 Dec 2022 20:27:08 +0000 (14:27 -0600)]
schema: add password configuration for ssh disk
Right now, ssh network disks are not usable. There is some basic support
in libvirt that is meant to support disk chains that have backing disks
located at ssh urls, but there is no real way for a user to configure a
ssh-based disk. This commit allows users to configure an ssh disk with
password authentication. Implementation will follow.
Jonathon Jongsma [Fri, 16 Dec 2022 23:10:49 +0000 (17:10 -0600)]
qemu: try to connect to nbdkit early to detect errors
When using nbdkit to serve a network disk source, the nbdkit process
will start and wait for an nbd connection before actually attempting to
connect to the (remote) disk location. Because of this, nbdkit will not
report an error until after qemu is launched and tries to read from the
disk. This results in a fairly user-unfriendly error saying that qemu
was unable to start because "Requested export not available".
Ideally we'd like to be able to tell the user *why* the export is not
available, but this sort of information is only available to nbdkit, not
qemu. It could be because the url was incorrect, or because of an
authentication failure, or one of many other possibilities.
To make this friendlier for users and easier to detect
misconfigurations, try to connect to nbdkit immediately after starting
nbdkit and before we try to start qemu. This requires adding a
dependency on libnbd. If an error occurs when connecting to nbdkit, read
back from the nbdkit error log and provide that information in the error
report from qemuNbdkitProcessStart().
$ virsh start nbdkit-test
2023-01-18 19:47:45.778+0000: 30895: error : virNetClientProgramDispatchError:172 : internal
error: process exited while connecting to monitor: 2023-01-18T19:47:45.704658Z
qemu-system-x86_64: -blockdev {"driver":"nbd","server":{"type":"unix",
"path":"/var/lib/libvirt/qemu/domain-1-nbdkit-test/nbdkit-libvirt-1-storage.socket"},
"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Requested export not
available
error: Failed to start domain 'nbdkit-test'
error: internal error: process exited while connecting to monitor: 2023-01-18T19:47:45.704658Z
qemu-system-x86_64: -blockdev {"driver":"nbd","server":{"type":"unix",
"path":"/var/lib/libvirt/qemu/domain-1-nbdkit-test/nbdkit-libvirt-1-storage.socket"},
"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Requested export not
available
After this change:
$ virsh start nbdkit-test
2023-01-18 19:44:36.242+0000: 30895: error : virNetClientProgramDispatchError:172 : internal
error: Failed to connect to nbdkit for 'http://localhost:8888/nonexistent.iso': nbdkit: curl[1]:
error: problem doing HEAD request to fetch size of URL [http://localhost:8888/nonexistent.iso]:
HTTP response code said error: The requested URL returned error: 404
error: Failed to start domain 'nbdkit-test'
error: internal error: Failed to connect to nbdkit for 'http://localhost:8888/nonexistent.iso]:
error: problem doing HEAD request to fetch size of URL [http://localhost:8888/nonexistent.iso]:
HTTP response code said error: The requested URL returned error: 404
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Adds the ability to monitor the nbdkit process so that we can take
action in case the child exits unexpectedly.
When the nbdkit process exits, we pause the vm, restart nbdkit, and then
resume the vm. This allows the vm to continue working in the event of a
nbdkit failure.
Eventually we may want to generalize this functionality since we may
need something similar for e.g. qemu-storage-daemon, etc.
The process is monitored with the pidfd_open() syscall if it exists
(since linux 5.3). Otherwise it resorts to checking whether the process
is alive once a second. The one-second time period was chosen somewhat
arbitrarily.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jonathon Jongsma [Mon, 21 Aug 2023 21:04:35 +0000 (16:04 -0500)]
qemu: Add Taint for nbdkit restart failure
Since the restart handler will trigger at an arbitrary time (when the
nbdkit process crashes, for instance), it's difficult to provide
feedback to the user if the restart is unsuccessful. Rather than just
relying on a warning in the log, taint the domain so that there will be
a slightly more user-visible notification.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jonathon Jongsma [Fri, 19 Aug 2022 22:21:52 +0000 (17:21 -0500)]
tests: add tests for nbdkit invocation
We were testing the arguments that were being passed to qemu when a disk
was being served by nbdkit, but the arguments used to start nbdkit
itself were not testable. This adds a test to ensure that we're invoking
nbdkit correctly for various disk source definitions.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
qemu: use nbdkit to serve network disks if available
For virStorageSource objects that contain an nbdkitProcess, start that
nbdkit process to serve that network drive and then pass the nbdkit
socket to qemu rather than sending the network url to qemu directly.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jonathon Jongsma [Thu, 18 Aug 2022 21:27:46 +0000 (16:27 -0500)]
qemu: pass sensitive data to nbdkit via pipe
Rather than passing passwords and cookies (which could contain
passwords) to nbdkit via commandline arguments, use the alternate format
that nbdkit supports where we can specify a file descriptor which nbdkit
will read to get the password or cookies.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
All users of virCommandSetSendBuffer() are using it to send sensitive
data to a child process. So, since these buffers contain sensitive
information, clear it with virSecureErase().
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Add xml to the private data for a disk source to represent the nbdkit
process so that the state can be re-created if the libvirt daemon is
restarted. Format:
This prepares encryption secrets and authentication secrets. When we add
nbdkit-backed network storage sources, we will not need to send
authentication secrets to qemu, since they will be sent to nbdkit
instead. So split this into two different functions.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jonathon Jongsma [Thu, 19 Jan 2023 19:16:01 +0000 (13:16 -0600)]
qemu: move qemuProcessReadLog() to qemuLogContext
This code can be used by the nbdkit implementation for reading back
filtered log data for error reporting. Move it to qemuLogContext so that
it can be shared. Renamed to qemuLogContextReadFiltered().
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Rather than hard-coding the nbdkit module directory, query the nbdkit
binary for the location to these directories. nbdkit provides a
--dump-config optiont that outputs this information and can be easily
parsed. We can also get the version from this output rather than
executing `nbdkit --version` separately.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
An object for storing information about a nbdkit process that is serving
a specific virStorageSource. At the moment, this information is just
stored in the private data of virStorageSource and not used at all.
Future commits will use this data to actually start a nbdkit process.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Add the virFileCache implementation for nbdkit capabilities to the qemu
driver. This allows us to determine whether nbdkit is installed and
which plugins are supported. it also has persistent caching and the
capabilities are re-queried whenever something changes.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
qemu: implement persistent file cache for nbdkit caps
Implement the loadFile and saveFile virFileCacheHandlers callbacks so
that nbdkit capabilities are cached perstistently across daemon
restarts. The format and implementation is modeled on the qemu
capabilities, but simplified slightly.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Since the libvirt documentation suggests to prefer GObject over
virObject, and since virObject is a GObject, change virFileCache to
allow GObjects as data.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
In order to add caching of the nbdkit capabilities, we will need to
compare against file modification times, etc. So look up this
information when creating the nbdkit caps.
Add a nbdkit_moddir build option to allow the builder to specify the
location to look for nbdkit plugins and filters.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
qemu: Add functions for determining nbdkit availability
In future commits, we will optionally use nbdkit to serve some remote
disk sources. This patch queries to see whether nbdkit is installed on
the host and queries it for capabilities. The data will be used in later
commits.
Signed-off-by: Jonathon Jongsma <jjongsma@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jim Fehlig [Mon, 18 Sep 2023 15:59:39 +0000 (09:59 -0600)]
libxl: Fix Domain-0 ballooning logic
When Domain-0 autoballooning is enabled, it's possible that memory may
need to be ballooned down in Domain-0 to accommodate the needs of another
virtual machine. libxlDomainFreeMemory handles this task, but due to a
logic bug is underflowing the variable containing Domain-0 new
target memory. The resulting huge numbers are filtered by
libxlSetMemoryTargetWrapper and memory is not changed.
Under the covers, libxlDomainFreeMemory uses Xen's libxl_set_memory_target
API, which includes a 'relative' parameter for specifying how to set the
target. If true, the target is an increment/decrement value over the
current memory, otherwise target is taken as an absolute value.
libxlDomainFreeMemory sets 'relative' to true, but never allows for
negative values by declaring the target memory variable as an unsigned.
Fix by declaring the variable as signed, which also requried adjusting
libxlSetMemoryTargetWrapper.
Signed-off-by: Jim Fehlig <jfehlig@suse.com> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
Erik Skultety [Wed, 13 Sep 2023 09:49:09 +0000 (11:49 +0200)]
ci: jobs.sh: Define and create SCRATCH_DIR for local executions
Running outside of GitLab will likely not have the variable set and
hence the execution would fail. To make sure we always start with a
clean scratch dir (which may or may not be the best thing), create it
with 'mktemp'. The main reason for a temporary directory is to ensure a
clean environment for the job every time run_integration function is
run. For repeated interactive use case, it is imperative that the
developer takes care of their environment.
Signed-off-by: Erik Skultety <eskultet@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
projects / libvirt.git / log