Anthony PERARD [Thu, 14 Mar 2024 17:50:23 +0000 (17:50 +0000)]
mfi-common: Rework toolstack-disk_format test matrix
There is 6 potenteil test of toolstack disk_format combination:
{xl,libvirt}-{raw,vhd,qcow2}. Commit f536e834f673 ("make-flight: Trim
the matrix of disk format flights") introduced a way to avoid testing
on every architecture and actually do only 6 tests accross all
arches, 3 on x86, 3 on armhf (no arm64 at the time).
But, commit 5c70735f177f ("fmtarches: Use dom0arches, not hardcoded
arch list") broke that. There's an extra space ' ' added at the
begining of the string in variable $fmtarches_outer which break the
algorithm. This mean that the combination of xl+raw is never tested
and both armhf and amd64 are testing the same combination of
toolstack+disk_format.
We could just remove the extra ' ', but that would mean 3 extra tests
on armhf and 3 extra tests on arm64. At least x86 would have a test
for each combination.
I'd like to avoid adding more arm* tests due to limited capacity, so
the new goal is to have the 6 combination of toolstack-disk_format
spread on both arm*, where each get to test all possible disk_format.
On x86, we are also going to tests all 6 combinations, which might all
happen on dom0=amd64 if we don't test dom0=i386, or they will be
spread on both kind of dom0.
To avoid too many changes in jobs on x86 with the current list of
jobs, we reverse the order of "i386" and "amd64".
Jobs changes on xen-unstable and most branches:
x86:
+ test-amd64-amd64-xl-raw
+ test-amd64-i386-libvirt-qcow2
arm*:
- test-armhf-armhf-libvirt-qcow2
- test-armhf-armhf-libvirt-raw
- test-armhf-armhf-xl-vhd
+ test-armhf-armhf-libvirt-vhd
+ test-armhf-armhf-xl-qcow2
+ test-armhf-armhf-xl-raw
Anthony PERARD [Wed, 13 Mar 2024 14:28:47 +0000 (14:28 +0000)]
make-fligh: Fix freebsd guest test test-id
Current freebsd as guest tests rely on the variable $qemuu_suffix, but
that one may or may not be set yet, and can't be rely upon. It isn't
set on the first iteration which call test_matrix_do_one(), with
xenarch=amd64 dom0arch=i386, but it is on the second call with
xenarch=amd64 dom0arch=amd64.
Only flights on branch linux-linus and linux-6.1 are changes (and the
disabled linux-next branch). This is because the first call of
test_matrix_do_one() is with dom0arch=amd64 as we skip "i386". On
those branches, we have this changes in jobs names:
Anthony PERARD [Tue, 20 Feb 2024 14:12:17 +0000 (14:12 +0000)]
ap-common: Switch to Linux 6.1 by default on Arm
The "linux-arm-xen" tree currently point to Linux 5.4.17 + 3 commits: 69117aaf07f7 ("xen/privcmd: replace kcalloc() by kvcalloc() when allocating empty pages") bca5bc0229a6 ("xen/privcmd: fix error handling in mmap-resource processing") f0f0e602f7c9 ("xen/privcmd: drop "pages" parameter from xen_remap_pfn()")
This stable version is old and doesn't build on Debian Bookworm.
The 3 extra commits are present in at least Linux 5.15, so there's
currently no reason to have a separate tree when we could use a stable
tree that osstest can update regulary.
So switch to 6.1. This will change the default Linux tree for all Arm
jobs that aren't on "linux-*" osstest branches.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> Acked-by: Stefano Stabellini <sstabellini@kernel.org>
Roger Pau Monne [Mon, 15 Jan 2024 10:05:11 +0000 (11:05 +0100)]
osstest: bump /boot partition to 1G
Increase the size of the boot partition, as arm64 initrd is already exhausting
all the space:
update-initramfs: Generating /boot/initrd.img-6.7.0+
gzip: stdout: No space left on device
Reported-by: Jan Beulich <jbeulich@suse.com> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Anthony PERARD <anthony.perard@citrix.com>
Anthony PERARD [Fri, 8 Dec 2023 15:38:35 +0000 (15:38 +0000)]
ts-kernel-build: Remove use of $XEN_TARGET_ARCH
osstest doesn't set $XEN_TARGET_ARCH, and this is a Xen build
variable, not a Linux build one. So, this code have been running with
an empty $XEN_TARGET_ARCH for a while which mean none of those
`setopt` has run in a while.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Anthony PERARD [Mon, 4 Dec 2023 16:46:24 +0000 (16:46 +0000)]
create_build_jobs: Enable X86_GENERIC for i386 kernel pvops jobs.
This is following a failure to build "arch/x86/xen/enlighten.c" in
build-i386-pvops jobs with linux 968f35f4ab1c ("Merge tag
'v6.7-rc3-smb3-client-fixes' of git://git.samba.org/sfrench/cifs-2.6")
in linux-linus branch.
Today the cpu type for a 32-bit x86 kernel is not specified in the
used kernel config, resulting in the M686 to be used.
Instead of using the M686 which isn't even a 64-bit cpu (thus not
capable to run a Xen guest), use the X86_GENERIC variant which is
more appropriate.
Suggested-by: Juergen Gross <jgross@suse.com> Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Anthony PERARD [Mon, 4 Dec 2023 16:46:08 +0000 (16:46 +0000)]
create_build_jobs: Set reset pvops_kconfig_overrides
Those two variables, and especially "pvops_kconfig_overrides", aren't
reset for the next loop of `for arch ...`. So when
"pvops_kconfig_overrides" is for "armhf", it is also set for "arm64"
as this is the next "arch" in the loop.
Avoid this by setting default values for the variables before the
`case` command.
This changes Linux arm64 builds which will not have "CONFIG_EXT4_FS=y"
anymore, but should get "CONFIG_EXT4_FS=m" instead from
`ts-kernel-build`. But resulting build still have "CONFIG_EXT4_FS=y",
so no change.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Anthony PERARD [Mon, 27 Nov 2023 16:11:11 +0000 (16:11 +0000)]
preseed_base: Remove reportbug-gtk if present
The Debian installer have started to install "reportbug-gtk" which
depends on "dconf-service" and "dbus-user-session" and "systemd".
And when we try to install "sysvinit-core" we are met with the
following error message:
> The following packages have unmet dependencies:
> dconf-service : Depends: default-dbus-session-bus or
> dbus-session-bus
So, remove that new package and its dependencies before moving on.
While here, add `set -x` to help debugging.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
ts-kernel-build: add kernel kconfig for the Arndale
The Linux commit 33629d35090f5ce2b1b4ce78aa39954c603536d5 has
removed the 'snps,dwc-ahci' compatible from the generic
AHCI-platform driver control module selected by
CONFIG_SATA_AHCI_PLATFORM.
A new driver, the DWC AHCI SATA platform driver is now implemented
and handles the above compatible when CONFIG_AHCI_DWC is selected.
The module is needed for the Arndale board to have the SATA controller
working, so enable the CONFIG_AHCI_DWC as additional kconfig parameter
in ts-kernel-build.
Signed-off-by: Luca Fancellu <luca.fancellu@arm.com> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Anthony PERARD [Thu, 22 Jun 2023 09:09:18 +0000 (09:09 +0000)]
ts-xen-build-prep: force use of git protocol v2
Recent version of QEMU (to be 8.1) started to use meson subproject to
clone extra repo. With the example of the subproject "dtc", they do a
shallow clone with a sha1. Meson end up running:
This command fails. I think the error message is something like "the
remote end hung up unexpectedly", on Debian Buster. A more useful
message with more recent version of git seems to be "couldn't find
remote ref".
If we allow git to communicate with the protocol v2, then the shallow
clone works. But git on buster still use v1 by default. Force it to
use v2.
This needs a git-cache-proxy version that can allow to switch to v2 of
the protocol. A bug is open upstream to track this change:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040476
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Anthony PERARD [Mon, 19 Jun 2023 14:37:14 +0000 (14:37 +0000)]
Use updated mirror for buster armhf, and update debian-installer
This reverts commit b838a9daeb3b ("production-config: Use a snapshot
for buster armhf")
Installation now fails with "Invalid Release signature", while
downloading the "Release" file of a repo.
But, using the main mirror for armhf boxes seems to work fine now.
To use the live mirror, we need to update the debian-installer as
there's a newer version of the kernel. Otherwise, installation fails
with the installer not able to find the disk.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Anthony PERARD [Mon, 22 May 2023 13:54:48 +0000 (13:54 +0000)]
ts-xen-build-prep: Install python3-venv for QEMU
Since QEMU commit 81e2b198a8cb ("configure: create a python venv
unconditionally"), a python venv is always created and this new package
is needed on Debian.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Current usage of Werror=switch-enum by default for libvirt builds out
of the git tree causes issues when new items are added to libxl public
API enums if those are used in a switch statement in libvirt code.
This leads to libvirt build failures for seemingly unrelated libxl
changes.
In order to prevent those errors from blocking the push gate, disable
Werror for libvirt builds when not in a libvirt specific flight.
The errors will be reported on the libvirt flight, and block the
pushes there. So the author of the changes in libxl is still expected
to send a fix to libvirt code. This is no ideal, but the other option
is to just disable Werror for all libvirt builds and let libvirt
developers fix the breakage when they notice it.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Reviewed-by: Anthony PERARD <anthony.perard@citrix.com> Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
---
Changes since v1:
- Use -Dwerror=false instead of -Dgit_werror=disabled
--- Cc: Ian Jackson <iwj@xenproject.org> Cc: Anthony PERARD <anthony.perard@citrix.com> Cc: Julien Grall <julien@xen.org>
Anthony PERARD [Thu, 19 May 2022 10:55:25 +0000 (11:55 +0100)]
ts-xen-build-prep: Install newer NASM version, to build OVMF
Recent versions of OVMF now need a version of NASM that is newer
than the one available on Debian oldstable/buster. They want to use
NASM 2.15.05 [1], which is available in Debian stable/bullseye. The
need to use a newer version started with d3febfd9ade3 ("MdePkg:
Replace Opcode with the corresponding instructions.").
There is no backport package available but the nasm package from
Debian Bullseye can easily be installed on Buster as it has few
dependencies and are already satisfied.
Or else all interrupts will get bound to (v)CPU 0.
This doesn't cause issues on small boxes, but boxes with a non-trivial
amount of CPUs can struggle without interrupts being balanced across
available vCPUs, as the number of vCPUs offered to dom0 matches the
number of physical CPUs.
For example sabro boxes (Xeon Silver 4114 x 2 sockets) would sometimes
report timeouts which seem to be solved by using irqbalance in dom0.
irqbalance is also available on Arm, so install unconditionally.
Reported-by: Jan Beulich <jbeulich@suse.com> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
Just disabling cron in rc.d is not enough. There's also anacron which
will get invoked during startup, and since apt-compat has a delay of
up to 30min it can be picked up by the leak detector if the test
finishes fast enough:
To prevent this disable anacron like it's done for cron.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
---
Changes since v1:
- Don't fail if the service is not present.
Ian Jackson [Tue, 14 Dec 2021 14:51:05 +0000 (14:51 +0000)]
daily-cron-email-real-*: Temporarily drop osstest-admin
Roger has agreed to take on osstest admin for the moment. Someone who
intents to take on the role long term will probably want to get CC's
of these flight reports, but it's fairly noisy. So for now, send them
only to the lists.
Signed-off-by: Ian Jackson <iwj@xenproject.org> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Roger Pau Monne [Thu, 25 Nov 2021 15:06:46 +0000 (16:06 +0100)]
ts-xen-install: enable timestamp on guests logs
Enable the timestamp feature of xenconsoled so guests logs have a time
reference. Can be helpful when debugging boot related slowness.
This requires using the XENCONSOLED_ARGS option and setting both the
log and the timestamp options. Note that setting XENCONSOLED_TRACE
will override any options in XENCONSOLED_ARGS, so they can not be used
in conjunction.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <iwj@xenproject.org>
Ian Jackson [Fri, 22 Oct 2021 15:21:27 +0000 (16:21 +0100)]
examination: skip memdisk on non-BIOS hosts, run per-firmware on x86
This is a combination of two changes:
ts-memdisk-try-append: skip memdisk test on non-BIOS hosts
make-flight: examine: Insist on -bios and -uefi tests on x86
This will let us skip ts-memdisk-try-append on non-bios platforms
without risking regression. It will also definitely spot
regressions which will occur on any uefi host.
standalone-generate-dump-flight-runvars reveals the changes are as
follows:
New jobs
test-amd64-i386-examine-bios test-amd64-i386-examine-uefi
test-amd64-amd64-examine-bios test-amd64-amd64-examine-uefi
added everywhere that has the corresponding plain job, namely
osstest
linux-*
xen-unstable
These jobs are just like the plain jobs, except that one of
,PropEq:Firmware:bios:bios
,PropEq:Firmware:bios:uefi
has been added to the end of all_hostflags.
Signed-off-by: Ian Jackson <iwj@xenproject.org> Release-Acked-by: Ian Jackson <iwj@xenproject.org>
Ian Jackson [Wed, 24 Nov 2021 14:34:47 +0000 (14:34 +0000)]
mg-repro-setup: Make ordinary alloc: work again
In e7febe5f6edc, we hosted an error earlier in the script. But in the
old location, OSSTEST_TASK would always be set if statictask was. In
the new one, that hasn't been done yet.
No release implications since it touches a by-hand utility only.
Fixes: e7febe5f6edc "mg-repro-setup: Promote an error test to before builds (nfc)" CC: Roger Pau Monné <roger.pau@citrix.com> Signed-off-by: Ian Jackson <iwj@xenproject.org> Release-Acked-by: Ian Jackson <iwj@xenproject.org>
Ian Jackson [Mon, 22 Nov 2021 11:51:01 +0000 (11:51 +0000)]
allow.all: Treat freeBSD memdisk-try-append failures as nonblocking
In practice this currently fails on some of our hosts. We don't seem
to have resources to investigate properly, and this will be blocking
recomissioning of hosts if we don't get it out of the way.
Signed-off-by: Ian Jackson <iwj@xenproject.org> Release-Acked-by: Ian Jackson <iwj@xenproject.org> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Ian Jackson [Wed, 27 Oct 2021 10:53:50 +0000 (11:53 +0100)]
ts-xen-build: Pass --enable if --disable found in usage, and v.v.
The existing code works in practice if the usage message always lists
the non-default, since the unlisted-in-usage options that would be
supported, but are elided, are in any case the default.
But configure might *compute* its defaults. In which case it will
list only one of them in the usage message. If the computed default
is not the same as the usual default (the one implied by listing the
opposite in the usage message) we would wrongly not pass the option.
So grep for both enable and disable.
Signed-off-by: Ian Jackson <iwj@xenproject.org> Reviewed-by: Juergen Gross <jgross@suse.com>
Juergen Gross [Tue, 26 Oct 2021 13:56:45 +0000 (15:56 +0200)]
osstest: explicitly set either enable or disable qemu-traditional
Instead of setting "--enable-qemu-traditional" or not, switch to
setting "--enable-qemu-traditional" or "--disable-qemu-traditional".
This avoids a latent bug in the disable case, as the availability
of the option is tested via grep, which will otherwise grep for
nothing.
Reported-by: Ian Jackson <iwj@xenproject.org> Signed-off-by: Juergen Gross <jgross@suse.com> Reviewed-by: Ian Jackson <iwj@xenproject.org> Release-Acked-by: Ian Jackson <iwj@xenproject.org>
Ian Jackson [Thu, 9 Sep 2021 15:46:27 +0000 (16:46 +0100)]
fmtarches: Use dom0arches, not hardcoded arch list
This will make us reallocate fmt tests when the arch list changes.
It's not ideal because it means tests jumping about across arches and
might let regressions go through but it's better than just dropping
them, and doing a better approach is complex.
Ian Jackson [Thu, 19 Aug 2021 12:46:21 +0000 (13:46 +0100)]
make-flight: Drop pvgrub (pvgrub1) tests
This is obsolete. In 2017 in 6abb2f113025 we wrote:
A consequence is that this test will test jessie forever. Eventually
jessie will rot so badly that this test fails and then we will no
longer be testing pvgrub1. Hopefully by then no-one will be using it.
This has now occurred.
I have verified with
OSSTEST_CONFIG=standalone-config-example eatmydata ./standalone-generate-dump-flight-runvars
that the only change is to drop jobs test-amd64-amd64-*-pvgrub.
CC: Jan Beulich <jbeulich@suse.com> CC: Juergen Gross <jgross@suse.com> CC: Jan Beulich <jbeulich@suse.com> Signed-off-by: Ian Jackson <iwj@xenproject.org>
Now (postgresql 11) start_value is in a different table. We don't
really care about it very much and mostly care about the last value,
so we fix that (and launder the for-db comparison dumps).
* psql transaction behaviour has changed so that now we want to
use the -1 option. This obviates a few BEGIN and COMMITs.
* SET implicitly starts a transaction and DROP and CREATE DATABASE
aren't transactional and now complain if they are run in a
transaction. So we must add COMMIT after SET.
Ian Jackson [Wed, 10 Feb 2021 16:41:12 +0000 (16:41 +0000)]
Disable updates for ssapshot.debian.org
security updates are a separate apt source.
The point of using snapshot is to avoid pulling in uncontrolled
updates, so we need to disable security updates.
The non-security SUITE-updates are disabled by this too. But
everything is on fire and I don't want another iteration while I
figure out the proper syntax for disabling only the security updates.
Ian Jackson [Tue, 9 Feb 2021 13:03:32 +0000 (13:03 +0000)]
mg-debian-installer-update: Honour redirect for dtbs
When using snapshots, we can get a redirect and then we don't
recurse. There doesn't seem to be a suitable option for wget, so do
this by hand before we call wget -m.
Ian Jackson [Fri, 22 Jan 2021 15:11:01 +0000 (15:11 +0000)]
make-flight: Stripy xenstored
Previously, we let the Xen build system and startup scripts choose
which xenstored to use. Before we upgraded to Debian buster, that
gave us C xentored tests on ARM. Since then, armhf and arm64 have
both had enough ocaml support and we haven't been testing C xenstored
at all !
Change this, by selecting between C xenstored and Ocaml xenstored
"at random". Actually, this is based on the job name. So the same
job in different branches will use the same xenstored - which helps
avoid confusion.
I have diffed the output of standalone-generate-dump-flight-runvars.
As expected, this addes a variable all_host_xenstored to every job.
To make sure we have enough diversity, I eyeballed the results. In
particular:
* The smoke tests now have 2 C and 2 Ocaml, one of each on
ARM and x86.
* XTF tests have 2 oxenstored and 3 C xenstored.
* The ovmf flight has one of each
* The seabios and libvirt flights look reasonably mixed.
Most other flights have enough jobs that I think things are diverse
enough without looking at them all in detail.
I think this lack of testing needs fixing for the Xen 4.15 release.
So after review I intend to push this to osstest pretest, and may
force push it even if shows regressions.
CC: Edwin Török <edvin.torok@citrix.com> CC: Andrew Cooper <Andrew.Cooper3@citrix.com> CC: Jürgen Groß <jgross@suse.com> CC: Wei Liu <wl@xen.org> Signed-off-by: Ian Jackson <iwj@xenproject.org> Release-Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Christian Lindig <christian.lindig@citrix.com>
Ian Jackson [Thu, 19 Nov 2020 16:55:48 +0000 (16:55 +0000)]
sg-report-flight: Actually look at retest flights (part 2)
To avoid going down ratholes (especially for jobs which reuse outputs
from their previous selves), the primary flight/job finder in
sg-report-flight does not recurse indefinitely through build jobs.
Instead, it restricts the build jobs investigated to those within the
same flight as the job which might be of interest.
As a result, retest jobs are, unfortunately, discarded at this stage
because we insist that the build jobs we find did use the tree
revision we are investigating.
Fix this by recursing into the corresponding primary flight too.
In the $flightsq->fetchrow loop that's $xflight.
For the primary flight, ie the first half of the UNION, that's just
the fligth itself. So there this has no change.
For the retest flights, it is the flight that all the build jobs refer
to. Despite the CROSS JOIN, this will be unique for any particular
"retest flight", because the query on the runvars insists that all of
the (at least some) buildjob runvars for f1 point to f0. Ie, f1 has
no build jobs and refers to f0 for build jobs; so it can't refer to
any other f0' in the cross join.
With this change, a -retest flight can now actually be used to justify
a push.
Ian Jackson [Thu, 19 Nov 2020 16:24:32 +0000 (16:24 +0000)]
sg-report-flight: Actually look at retest flights (part 1)
The existing approach does not find retest flights. This is because
it starts by looking for flights whose runvars say they built the
version in question, and then looks to see if they contain the
relevant job.
Retest flights don't contain build jobs; they reuse the builds from
the primary flight.
Rather than making a fully general recursion scheme (which would
involve adding another index so we could quickly find all flights
which refer to this one), we add a one-level recursion.
This recursion is restricted to the flights named on the command line.
This means it takes nearly no time (as opposed to searching the whole
db for things that might be relevant - see above re need for an
index).
We filter the command line flights, looking ones which refer to the
only the primarily found flights as build jobs.
projects / osstest.git / log