Wei Liu [Thu, 2 Nov 2017 15:15:04 +0000 (15:15 +0000)]
make-flight: guest should use jessie to test pvgrub
stretch has 64bit feature enabled for ext4, which pvgrub can't cope.
We want to continue to test pvgrub, so specify jessie in the guest
suite field.
A consequence is that this test will test jessie forever. Eventually
jessie will rot so badly that this test fails and then we will no
longer be testing pvgrub1. Hopefully by then no-one will be using it.
CC: Juergen Gross <jgross@suse.com> Signed-off-by: Wei Liu <wei.liu2@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Adjust commit message slightly.
Wei Liu [Fri, 20 Oct 2017 14:35:06 +0000 (15:35 +0100)]
ts-guests-nbd-mirror: make it work with stretch
On the server side, only add oldstyle= and port= on wheezy and jessie.
stretch doesn't support or need those anymore.
On the client side, generate new style configuration file.
Reorder nbd-client setup a bit. Install it first, then write our own
configuration file, then start it. This stops dpkg asking what to
do regarding configuration files.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v3: invert some tests, rearrange client setup code.
v4: Fix commit message grammar.
Wei Liu [Tue, 17 Oct 2017 16:10:10 +0000 (17:10 +0100)]
ts-debian-fixup: merge origin extra= to our own if necessary
The original extra= was not removed, so there were two extra= in the
resulting config file.
It wasn't a problem for xl because the second extra= took precedence.
However libvirt tests would only pick up the first extra= so they
worked by chance.
Fix this issue by merging the original. If there isn't already extra=
in $cfg, use our own.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v3: handle situation when no extra= is in $cfg
Ian Jackson [Tue, 2 Apr 2019 14:56:57 +0000 (15:56 +0100)]
power: Fix uninitialised variable warning
In
power: Record approach used for power cycles in runvars
we introduced a reference to $r{$rv} which might be undef,
resulting in this:
Use of uninitialized value in concatenation (.) or string at Osstest/TestSupport.pm line 1069.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 25 Feb 2019 15:46:52 +0000 (15:46 +0000)]
jessie: Disable use of security.debian.org
We have about a 10% failure rate of a problem where the symptoms are
that the test box fails to get some thing from security.debian.org.
The apt-cacher-ng logs show the relevant test box's ip address
fetching the file that it is supposed to. But, it is possible that
there are different timeouts, so that does not mean the problem is
inside the colo.
Fetching from other apt sources, notably the main Debian archive and
snapshot.d.o, do not seem to be affected.
Specifically, I searched the logs for the last 1000 host install steps,
and looked for the failures, with the following rune:
select flight,job,logfile,started from (select *, (select val from runvars r where r.job=steps.job and r.flight=steps.flight and r.name='host') from steps where testid like 'host-install%' and flight>130000 order by finished desc limit 1000) sub where status='broken';
I then used these runes to correlate that with the syslogs from the
installer:
perl <~/t -ne 'use strict; s/^ *//; my ($flight,$job,$logf) = split / +\| +/; next unless $flight =~ m/^\d+$/; my $f = "$flight/$job/?.ts-syslog-server.log"; my @y = glob $f; print $_, "\n" for @y;' >~/u
xargs <~/junk/u egrep -L 'Failed to fetch http://security\.debian\.org.*Connection failed'
The only logs which did not mention that error message were three
failed jobs on the same host, joubertin1, which seems not to be
rebooting reliably.
So I think this is a problem with the security.debian.org CDN.
For now, disable security updates entirely. We don't really care
about the security patch status of test boxes anyway. Hopefully this
will cause the system to become reliable again.
CC: Juergen Gross <jgross@suse.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 13 Feb 2019 17:07:23 +0000 (17:07 +0000)]
backports snapshot: Use 20190206T211314Z for jessie-backports
Some time on 2019-02-07, Debian removed linux-base from
jessie-backports. This caused everything to break: apt wasn't happy
to get linux-base from jessie-security (because of our -t
jessie-backports, probably) and that meant there was no linux-base
suitable for linux-image-4.9.x on arm64. We ended up trying to
boot the installed system with 3.16, which does not work on our two
SoftIron arm64 test boxes.
Also, jessie-backports about to be completely removed.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 25 Jan 2019 12:04:21 +0000 (12:04 +0000)]
power: ssh: Wait for the target to appear to go down
When we `power on' with the ssh method, we actually run ssh reboot.
On some systems (notably, FreeBSD) the kernel does not simply reboot
immediately even with the runes we provide here, ie for FreeBSD
reboot -nq
Eg, I have seen reboots with several messages like this:
Jan 25 14:17:59.100044 Waiting (max 60 seconds) for system thread `bufspacedaemon-2' to stop... done
This can result in the ssh method failing spuriously, because the
`power on' appears to complete while the host is still up in the
previous environment. In one of my test runs I saw an ssh to the host
succeed, and print the uptime (of the existing environment), between
the reboot command being issued and the host actually rebooting.
So, wait (up to just over a minute) until the host does not respond to
ping. (target_await_down runs ping -c 5.)
Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 24 Jan 2019 18:25:24 +0000 (18:25 +0000)]
power: ts-freebsd-host-install: Use power_reboot_attempts
We look at the installer environment uptime, to
| check that this is the installer environment
as requested by the comment
| in particular $await must only succeed if the host really did
| reboot into the boot environment that $await expects.
near the top of power_reboot_attempts
Reviewed-by: Roger Pau Monné <roger.pau@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 25 Jan 2019 12:08:41 +0000 (12:08 +0000)]
power: ssh: Fix handling of $delay
The script fragment contains a reference to $delay which is a perl
variable, not a variable in the script fragment. We therefore need to
not ''-quote the script.
Without this, the ssh method will often fail spuriously: the exiting
parent (which will signal success back to the osstest controller)
races with the attempt to reboot.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 22 Jan 2019 17:58:54 +0000 (17:58 +0000)]
power: power_reboot_attempts: Honour an $approach_re
The semantics are slightly different here: not specifying it means to
try everything rather than only the hardest. But the effect is
similar: not specifying $approach_re means we must succeed.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 22 Jan 2019 14:44:52 +0000 (14:44 +0000)]
power: Provide `ssh' power method
This is not really a power method but it can pretend to be one. On
power off, it does nothing. On power on it logs into the host to ask
it to do a hard reboot.
This is rather best effort, but it is eminently suitable for our new
approach/attempts arrangements because those will try another approach
if ssh didn't work.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 22 Jan 2019 15:50:00 +0000 (15:50 +0000)]
power: New ILOM/PDU arrangements - try just IPMI
We honour two new host properties PowerPDU and PowerILOM, in
preference to PowerMethod. The semantics are going to be properly
documented in a later patch, but, briefly:
If only one of these is supplied, it works like PowerMethod, except
that `nest' is applied by default.
If both are supplied, we make two approaches: one is just ILOM. The
other is to use ILOM and PDU together, with pause in between, and with
try_off applied to ILOM.
The current configuration in Massachusetts is, for hosts with IPMI, to
provide a PowerMethod specifying ad hoc to use PDU and then IPMI, and
also to provide both PowerPDU and PowerILOM.
The overall result of this patch, with that configuration, is to avoid
using the PDU at all if an IPMI-requested reboot is successful. This
should significantly reduce the number of hard power cycles for hosts
with IMPI.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 22 Jan 2019 17:01:02 +0000 (17:01 +0000)]
power: Provide `try_off' pdu method; deprecate ipmi_try
We are going to want to use this magically, in our new approach. Make
a general version, and deprecate ipmi_try (which will be obsoleted by
the new approach and which has probably not been used very much).
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
This new variable contains a list of different approaches to try.
* Move the meat of power_state into power_approach_invoke.
* power_state now looks for a single approach to try.
* The default for power_state is to pick the last approach in
the list, which by definition is supposed to be the most reliable.
* Currently there will only be one approach, `Only'.
No overall functional change other than to log messages.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 17 Jan 2019 15:20:11 +0000 (15:20 +0000)]
line wrapping: Use tmp/$flight.report in two extra places
The output from sg-report-flight might in principle contain long
lines, although this is not expected. So we are going to want to feed
it through the new cr-fold-long-lines.
Rather than piping, we are going to keep a copy of the .report file,
like is done in mg-execute-flight. So for now, just make that change.
No overall change other than to leave behind the tmp/$flight.report
file. It will be tidied up by the usual cleanup processes.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 17 Jan 2019 15:17:22 +0000 (15:17 +0000)]
line wrapping: Provide cr-fold-long-lines script
This is a reversible transformation which usually just introduces a \
where it splits lines.
We are going to use this to wrap the lines in our emails. SMTP has a
999-byte length limit (including a CR-LF pair). This can cause our
emails to go astray. We don't really want our messages to be q-p or
base64-encoded if we can avoid it, and MTAs don't do that anyway (so
we would have to organise it). So instead, we will simply wrap any
long lines that occur.
This transformation is not suitable for headers, but we don't intend
or want to generate long lines which would need further wrapping. (A
reversible transformation suitable for headers would be quite ugly and
would only be right for a subset of headers anyway...)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 16 Jan 2019 11:32:06 +0000 (11:32 +0000)]
ts-livepatch-run: Fix erroneous $$ in double-check
The doubled $s here are simply a mistake. The result is to make this
test ineffective, since `$$file' means `the value of the variable
whose name is in the variable $file', which here will never exist.
This produces a `Use of uninitialized value' warning and substitutes
the empty string, so overall we test the existence of the directory.
The missing check is not of much consequence since this check is not
really expected ever to fail, and if it does, some actual test
execution would fail due to the missing file.
So overall I think the only change is to log output.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Ian Jackson [Wed, 16 Jan 2019 11:30:49 +0000 (11:30 +0000)]
ts-livepatch-run: Print a message about expected failures
target_cmd_output_root_status prints the command exit status. If that
was a failure and the failure was as expected, this can be confusing
to readers who do not know that this is a possibility. So print a
message about it.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Ian Jackson [Mon, 13 Aug 2018 14:54:25 +0000 (15:54 +0100)]
production-config: Temporarily drop arm64
Both our arm64 boxes are out of commission again.
I have filed a ticket to have the recently failed box investigated.
And I am working on the stretcb upgrade which will bring our
Thunder-X into service.
But for now this is the best we can do to unblock everything.
CC: Julien Grall <julien.grall@arm.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Mon, 19 Nov 2018 16:24:42 +0000 (16:24 +0000)]
production-config*: Update to fishdescriptor with chroot bugfix
Deployment note: I have copied this binary to the images directory in
Cambridge and Massachusetts. The corresponding patch to chiark-utils
is on its way to my upstream hat.
CC: George Dunlap <george.dunlap@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 19 Nov 2018 16:19:44 +0000 (16:19 +0000)]
ts-depriv-audit-qemu: Create complete /run/user in appropriate root
* Use mkdir -p, rather than trying to only create /run/user/$uid.
That helps if /run and/or /run/user do not exist, as they do in
libxl-made chroots with recent libxl (which gets qemu to chroot).
* Do all of this in the root directory of the qemu process, not our
own root directory. So it works if qemu is chrooted.
CC: George Dunlap <george.dunlap@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Actually use $qpid rather than hardcoding 8123 !
squash! ts-depriv-audit-qemu: Create complete /run/user in appropriate root
Ian Jackson [Mon, 5 Nov 2018 18:31:04 +0000 (18:31 +0000)]
ts-xen-build: Force reliance on ipxe tarball
xen.git/tools/firmware/etherboot/Makefile tries to get a tarball from
xen-extfiles first and if that fails, tries cloning from ipxe.org.
ipxe.org is sometimes down (or half-down) and when that happens
without a tarball the build breaks and is hard to fix.
We would like, therefore, to ensure that the tarball is always made
before the commit which refers to it is in xen.git#master.
osstest has no knowledge of ipxe as a separate thing and just lets
xen.git have whatever version is in Config.mk. So osstest never needs
to specify particular ipxe version by hash, or the like.
So simply make osstest rely on the tarball existing, by having it
specify a completely invalid URL for the git clone. This will detect
attempts to update IPXE_GIT_TAG without making a corresponding
tarball.
CC: Paul Durrant <paul.durrant@citrix.com> Acked-by: Wei Liu <wei.liu2@citrix.com>
Ian Jackson [Thu, 1 Nov 2018 13:02:50 +0000 (13:02 +0000)]
mfi-common: Enable livepatch tests in 4.10 and 4.11 as intended (!)
The pattern xen-4.1* matches xen-4.10-testing, xen-4.11-testing, etc.
The right approach is to state the excluded branch names explicitly
as is done elsewhere.
Resulting changes (standalone-generate-dump-flight-runvars):
* new jobs: test-amd64-{amd64,i386}-livepatch
* runvar set: enable_livepatch=true in build-ARCH, build-ARCH-xsm
* all of the above in branches: xen-4.10-testing xen-4.11-testing
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Andrew Cooper <andrew.cooper3@citrix.com>
Ian Jackson [Thu, 1 Nov 2018 13:25:50 +0000 (13:25 +0000)]
mfi-common: Do not set enable_livepatch on build-*-prev
branch_wants_livepatch checks the current Xen version. We don't want
to try enabling livepatch on trees that don't have it. And for the
-prev build we care about forward migration etc., not about
livepatching the previous version.
Resulting changes (standalone-generate-dump-flight-runvars):
runvar removed: enable_livepatch=true
in jobs: build-amd64-prev build-i386-prev
in branches: xen-4.8-testing xen-4.9-testing xen-unstable
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Andrew Cooper <andrew.cooper3@citrix.com>
Ian Jackson [Fri, 26 Oct 2018 10:57:42 +0000 (11:57 +0100)]
production-config-cambridge: Use squid cache, not local apt proxy
debian.uk.xensource.com is no more (and the service name seems to have
been discarded). There is repo.citrite.net instead but it is a
proprietary appliance which - although it claims to be able to handle
apt repositories - is not compatible with debian-installer; at least,
jessie's d-i.
Swap to ftp.uk.debian.org (and rely for caching on the squid cache
which we have for non-apt stuff already). That works well enough for
stable suites, at least.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 24 Oct 2018 10:16:32 +0000 (11:16 +0100)]
mfi-common: Drop freebsd builds on non-freebsd flights
Something is wrong with the freebsd repositories. This causes freebsd
host installation to fail. This problem has persisted for a while now
and is blocking other work.
For now, drop these. The jobs being dropped are
build-amd64-xen-freebsd
build-amd64-xen-xsm-freebsd
in the branches
osstest
xen-unstable
(verified with standalone-generate-dump-flight-runvars).
CC: Roger Pau Monné <roger.pau@citrix.com> CC: Wei Liu <wei.liu2@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Roger Pau Monné <roger.pau@citrix.com>
Ian Jackson [Wed, 24 Oct 2018 10:24:36 +0000 (11:24 +0100)]
sa-init-db: Fix argument parsing
It claimed to have a default db name, but it didn't. Rework the
option parsing to make the usage message true. (That saves me typing
sa-init-db standalone
and getting bizarre errors as sqlite tries to treat the shell script
`standalone' as if it were a database file.)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 19 Sep 2018 10:52:37 +0000 (11:52 +0100)]
make-flight: Test with `sched=credit'
The default in xen-unstable#staging is soon to be sched=credit2. That
leaves osstest not testing credit1 at all.
Add a job that does this. It's a copy of the -credit2 job. (It's
a pretty nugatory test but it will probably do.)
We call the test -credit1, although the Xen parameter is =credit.
I have verified with standalone-generate-dump-flight-runvars that the
effect is to add the appropriately mutated copy of the credit2 job to
every flight. The mutations are solely the sched= parameter and the
job name.
CC: George Dunlap <george.dunlap@citrix.com> CC: Dario Faggioli <dfaggioli@suse.com> CC: Jan Beulich <JBeulich@suse.com> CC: Andrew Cooper <andrew.cooper3@citrix.com> CC: Stefano Stabellini <sstabellini@kernel.org> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Tim Deegan <tim@xen.org> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Reviewed-by: Wei Liu <wei.liu2@citrix.com>
Jim Fehlig [Wed, 5 Sep 2018 21:43:12 +0000 (15:43 -0600)]
libvirt: Install GnuTLS for libvirt builds
Since libvirt commit 60d9ad6f GnuTLS is required to build libvirt. The
various libvirt build tests in osstest began failing after the commit
hit libvirt.git master. Adding libgnutls28-dev to the list of packages
needed to build libvirt will fix the currently broken builds.
Signed-off-by: Jim Fehlig <jfehlig@suse.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
elbling boxes uses the mfi driver and have the hard disks in
passthrough mode, which means the mfi driver will expose them as
mfisyspd? instead of mfid?. In order for the installer to detect such
disks add mfisyspd0 to the list of disks to probe.
This should fix the host-install issues reported on elbling boxes:
Suggested-by: Ian Jackson <ian.jackson@eu.citrix.com> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v2:
- Only add FreeBSD Xen buildjobs to the xen-* branches.
Changes since v1:
- Fix enabling of FreeBSD Xen buildjob based on branch.
- Introduce a helper to add the FreeBSD Xen build jobs.
- Introduce the ts-xen-build-freebsd wrapper around ts-xen-build for
FreeBSD.
- Introduce a create_xen_build_job helper.
osstest: introduce a helper to create Xen build jobs
This is currently a non-functional change, since no new jobs are
added, and the existing ones should stay unchanged. Runvars of all
flights remain unchanged.
The purpose of the helper is to allow adding different Xen build jobs,
with likely different runvars in the future.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Roger Pau Monne [Fri, 29 Jun 2018 08:18:04 +0000 (10:18 +0200)]
osstest: set the make command to use for xen-build
The default make on FreeBSD is the BSD make, and Xen requires the GNU
make in order to build. Set the make command based on the OS for the
Xen build.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v1:
- Use gmake for all BSDs.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v2:
- Fix hosts flight to not require BIOS firmware.
Changes since v1:
- Fix nonbreaking space.
- Fix long line.
Roger Pau Monne [Fri, 29 Jun 2018 11:45:09 +0000 (13:45 +0200)]
osstest: remove duplicate set_freebsd_runvars
The set_freebsd_runvars helper in mfi-common is a superset of the
original function present in make-freebsd-flight, and will attempt to
fetch the last anointed FreeBSD build as a last resort option if no
FreeBSD build is signaled from the FreeBSD env vars. There's no reason
to have this duplication, since the set_freebsd_runvars in mfi-common
is perfectly suitable to be used by make-freebsd-flight.
This duplication was wrongly introduced by d36a7d892f by adding a
set_freebsd_runvars to mfi-common without removing the original
function in make-freebsd-flight.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v1:
- Add commit message.
Ian Jackson [Tue, 17 Jul 2018 14:50:35 +0000 (14:50 +0000)]
dummy flight generation: Tolerate lack of repos
In particular
OSSTEST_CONFIG=standalone-config-example eatmydata bash -x ./standalone-generate-dump-flight-runvars
should not require ~/osstest-repos to exist.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
projects / osstest.git / log