Anthony PERARD [Wed, 15 Jul 2015 16:27:57 +0000 (17:27 +0100)]
Create a flight to test OpenStack with xen-unstable and libvirt
This patch should create a flight "openstack-nova", with those jobs:
build-amd64
build-amd64-xsm
build-amd64-pvops
build-amd64-libvirt
test-amd64-amd64-devstack
test-amd64-amd64-devstack-xsm
build-armhf
build-armhf-libvirt
build-armhf-pvops
build-armhf-xsm
test-armhf-armhf-devstack
test-armhf-armhf-devstack-xsm
About the runvars revision_* of test-*-*-devstack:
only REVISION_OPENSTACK_NOVA is set, the others are unset.
Empty revision_* runvar would clone the default branch, which should
be master for every openstack repos.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Anthony PERARD [Thu, 6 Aug 2015 11:33:38 +0000 (12:33 +0100)]
ts-openstack-tempest: Run Tempest to check OpenStack
This script runs the OpenStack integration test suite, Tempest.
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 18 Apr 2017 13:09:38 +0000 (13:09 +0000)]
sg-run-job: Honour skip_testids runvar
This works like truncate_testids, except that the steps which match
are never run (truncate_testids skips steps *after* matching steps,
and also doesn't skip some things like log capture).
If the programmed testid for a step ends in `(*)' to request
substitutions of the stepno, the skip matching takes place against the
unsubstituted value (since it happens before a stepno is allocated).
There is no way to skip only some of a set of steps whose testids are
distinguished only by stepno.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 18 Apr 2017 12:39:29 +0000 (12:39 +0000)]
truncation: Support globs, and multiple patterns
Rename the variable to truncate_testids. It contains glob patterns as
for Tcl `string match', space-separated.
Adjust the two places which set it: cs-bisection-step (which needs to
quote any special characters) and mg-repro-setup (which does not
really process the value, but ought to be able to cope with
space-separated lists.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 10 Apr 2017 15:33:42 +0000 (15:33 +0000)]
mg-repro-setup: New script for setting up repros automatically
Given a previously failed job, this will:
* Create a flight for the repro attempt
* (Optionally) allocate a host to the user's personal task
* (Optionally) wipe the host
* Install the version of Xen and Linux used by that flight
* Run the job until the specified step finishes
* Email the user
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 10 Apr 2017 15:31:26 +0000 (15:31 +0000)]
mg-allocate: Provide --stdout-output
This prints RESTYPE/RESNAME/SHAREIX to stdout, after allocation is
successful (outside the db retry loop). This means that a
programmatic caller can now tell what was allocated, even if the spec
was complex.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 6 Apr 2017 17:06:10 +0000 (17:06 +0000)]
proxy config: Add ability to install MITM TLS cert
We want things like build jobs to be able to download things via
https. But we want them to be cached. To this end, we are having our
squid treat CONNECT as a request to MITM the TLS connection.
But this means that clients will see squid's cert, not the real one.
So placate them by installing the cert on each test box.
(The squid becomes part of the TCB for our coverity upload password,
but that is fine.)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 20 Mar 2017 15:17:31 +0000 (15:17 +0000)]
cs-adjust-flight: Work properly on db transaction retry
The variable @changes is eaten during processing. So on db retry,
there would be changes missing. (When creating a new flight, this
would often mean a flight with no jobs, which cannot be executed.)
CC: Roger Pau Monné <roger.pau@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 22 Feb 2017 11:45:58 +0000 (11:45 +0000)]
html output: ReportHtmlUnpubBaseUrl: New config variable
This allows us to generate private urls for unpublished flights. This
will be useful for links from the ongoing flights summary - since,
naturally, none of those logs are published yet.
The URLs will be of more limited use; the exact usefulness will depend
on the deployment.
If no separate Unpub url is specified, simply use the Pub one.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 26 Jan 2017 16:26:22 +0000 (16:26 +0000)]
bisection: Really avoid flail
The flail detector needs to search for flights which will look like
the one it is creating, when this one is done. The intended blessing
always lacks `-bisect' so is no use.
We can use the last blessing for this: both cr-try-bisect and
cr-try-bisect-adhoc pass the same value for the last blessing and for
the 2nd parameter to perhaps_bisect_step_core (which is passed to
sg-execute flight to properly bless the flight when it's done).
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 18 Jan 2017 16:13:21 +0000 (16:13 +0000)]
d-i: backports: When using backports kernel for a host, install linux-base
Newer kernels may need a newer linux-base. This call has the happy
side effect of adding backports to its sources.list, which is also
generally needed.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Wed, 18 Jan 2017 15:49:56 +0000 (15:49 +0000)]
netboot: UEFI grub: Do it more like pxe
* Abolish setup_grub_efi_bootcfg and use setup_netboot_bootcfg
instead. The latter function now expects to be given a config file
for the host's bootloader. For a UEFI host, that's a grub config
file, which it will write to the configured NetGrub location for the
specific host.
* Consequently, we no longer copy an actual grub image around every
time we do a bootloader setup. Instead, we expect that the tftp
server will be configured to point relevant hosts at a suitable
fixed image, which will go to our config file.
* Also, consequently, write the NetGrubTemplate path when we want to
setup the bootloader, rather than the ad-hoc grub-%ether%.
* Update mg-netgrub-loader-update jessie so that it writes the
NetGrubTemplateReal from the config into the grub.cfg so that the
grub image will read that file and then read the file created by
setup_netboot_bootcfg.
* Document this in its head comment.
Currently this new mg-netgrub-loader-update is untested because jessie
i386 grub (which is what we would run on the osstest controllers) does
not understand the arm64-efi target. But the new Netgrub.cfg location
is in use.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Wed, 18 Jan 2017 15:26:16 +0000 (15:26 +0000)]
netboot: Introduce TftpNetGrub*
* When expanding netboot filenames, look for NetGrub config keys
rather than Pxe filenames. Currently this only affects mg-hosts
because the files are created by setup_grub_efi_bootcfg, and the
files are of course referred to only by the actual netboot grub
image (which may be made by mg-netgrub-loader-update: neither of
which honour these variables.
* Update the documentation, the default configs, and the
production-config-*.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 22 Dec 2016 13:43:01 +0000 (13:43 +0000)]
ts-xen-install: Pass `noreboot' to Xen
This prevents Xen from rebooting the host, if Xen crashes.
This reboot serves no function in osstest, since a crashed host will
be automatically power cycled to recover it. (Firstly, during log
collection, a renewed attempt to boot from the hard disk; then, during
the next test, netboot to wipe the machine to reinstall it.)
But the reboot does make logs more confusing, and we suspect that the
reboot loops which can occur (eg if the version of Xen and Linux being
tested always crashes on boot) might be implicated in our test boxes
occasionally forgetting their boot order.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 20 Dec 2016 17:15:26 +0000 (17:15 +0000)]
db retry: Use HandleError and exceptions to detect when to retry
It appears that sometimes, $dbh->state could be overwritten before
$mjobdb->need_retry got to run. $dbh->err and $@ would still be
right. I have not been able to explain this; I suspect that there is
something exciting going on in the eval exception trapping.
To try to isolate the problem, I developed this different approach: we
use the HandleError database handle feature, to cause all the
retriable errors to be thrown as a dedicated exception class.
We can then simply test ref $@. (We don't care about subclassing
here. And we don't want isa because $@ might just be a string.)
This is, in general, more reliable, as it cannot treat any other
failures as db retry failures.
Osstest::Executive and Osstest::JobDB::Executive become slightly more
intertwined with this patch.
Sadly this does not seem to completely eliminate the problem. It does
allow us to present clearer evidence of problems in the underlying
DBI, DBD or PostgreSQL layers...
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 20 Dec 2016 17:29:57 +0000 (17:29 +0000)]
db retry, bisect: Cache build reuse investigations
If we previously searched for builds to reuse, trust our previous
answers. We will only have seen data from committed transactions and
we will only have looked at jobs in completed flights, which won't
have changed.
So any previously reuseable build is still reuseable. (Unless its
stash check failed, in which case we might want to search again if we
do a db retry.) We don't care much about missing any
recently-finished jobs - there is a much bigger and unavoidable race
there anyway, where multiple bisections on different branches may
choose to pointlessly rebuild the same thing at the same time.
Not doing this search over and over again is important because it is a
very wide ranging search, which will often cause database transaction
serialisation errors. Without some caching here, we may never
converge.
In principle we could do this another way: we could make a readonly
transaction which did all the searching. But that's a more awkward
way to organise the code because our search uses a temporary table
which we then construct the flight from.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 20 Dec 2016 17:21:16 +0000 (17:21 +0000)]
db retry, bisection: Reset %jobs_created on db retry
%jobs_created is used for memoisation while populating the destination
flight. We need to reset it when we restart flight construction,
because those jobs were created in the discarded transaction.
Otherwise we could create a flight with missing jobs.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
projects / people / aperard / osstest.git / log