Ian Jackson [Thu, 23 Jul 2015 16:31:52 +0000 (17:31 +0100)]
cs-bisection-step: Fix memoisation of search_compute_length_at
There was a half-implemented memoisation. Memoisation is necessary
because otherwise the algorithm is exponential in the commit history
depth (with base equal to the commit parent fanout).
Sort this out:
* Break out the actual computation into a new
search_compute_length_at_intern
* Deleting the individual memo assignments, which incidentally
means we no longer miss an (unimportant) one.
* Actually having the new memoising function search_compute_length_at
check $n->{LengthAt} (this is the bugfix).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
To reproduce a recent bisection problem I needed to exclude not just
all flights after a certain number, but also one earlier flight. So I
invented this option (and associated yaks).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 23 Jul 2015 17:32:16 +0000 (18:32 +0100)]
Flight restriction: Change implementation of --max-flight
Abolish $maxflight. All the users outside Osstest::Executive have
been eliminated, so this is fine. Replace it with
$restrictflight_cond, which can accumulate multiple conditions.
There is a minor functional change: when multiple --max-flight options
are specified, _all_ of them take effect (effectively using the lowest
value). That option is not used in production, and I don't expect
people elsewhere to be passing multiple different such options.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 23 Jul 2015 16:54:53 +0000 (17:54 +0100)]
Flight restriction: Make report_blessingscond use implicit $maxflight
We have $maxflight in Osstest::Executive now, set appropriately.
Use that in report_blessingscond and all its callers including
report_find_push_age_info and hence in mg-all-branch-statuses and
sg-report-flight and sg-report-job-history.
No functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 23 Jul 2015 17:01:22 +0000 (18:01 +0100)]
Flight restriction: Update cs-bisection-step
Use restrictflight_arg and restrictflight_cond.
This entails replacing $maxflight_cond (which is empty or contains a
series of texts each starting with AND) with $restrictflight_cond
(which is actually an expression, and might be just "TRUE").
No functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 23 Jul 2015 16:38:27 +0000 (17:38 +0100)]
Flight restriction: Introduce restrictflight_arg and restrictflight_cond
Provide the new functions which we will eventually use everywhere.
* Introduce restrictflight_arg which can be used in an argument loop.
* Introduce restrictflight_cond which generates a SQL condition.
* Have restrictflight_cond's value include surrounding ( ) for safety.
* Use the above in sg-report-host-history, renaming $maxflightcond.
No overall functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 23 Jul 2015 17:20:33 +0000 (18:20 +0100)]
Flight restriction: Move $maxflight to Osstest::Executive
We are going to move the interpretation of --max-flight into
Osstest::Executive. Start this by making $maxflight shared between
Executive and the main program. Later, we will make it no longer
exported.
There are a number of places in Executive where a `my' variable
$maxflight shadows the newly moved global. These will be got rid of
later, too.
No functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Wed, 8 Jul 2015 12:31:02 +0000 (13:31 +0100)]
make-distros-flight: Use ftp.debian.org directly
The local proxy seems to serve stale packages for Jessie etc, I blame
the intercepting cache on the way out of our network, similar to b5f15136900d "mg-debian-installer-update: workaround caching proxies",
except it is between the apt-cache and the world not the osstest vm
and the world.
Since the netboot kernel+initrd are reasonably small, these flights
are infrequent and they are intended to test the current upstream
version I think this is tollerable.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 8 Jul 2015 12:31:01 +0000 (13:31 +0100)]
Add testing of file backed disk formats
xen-create-image makes this tricky to do since it is rather LVM
centric. Now that we have the ability to install from d-i it's
possible to arrange fairly easily that they use something other than
a phy backend over a bare LVM device.
Here we add support to the test script and infra and create a bunch of
new jobs testing the cross product of {xl,libvirt} x {raw,qcow2,vhd}.
A disk format of "raw" means a raw backing file, where as "none" (the
default) means to continue to use the base LVM device.
The test scripts are modified such that when constructing a domain
with a diskfmt runvar specifeies a file backed disk format (i.e. not
"none"):
- the LVM device is slightly enlarged to account for file format
headers (1M should be plenty).
- the LVM device will have an ext3 filesystem created on it instead
of being used as a phy device for the guest. Reusing the LVM volume
in this way means we don't need to do more storage management in
dom0 (i.e. arranging for / to be large enough, or managing a
special "images" LV)
- the relevant type of container is created within the filesystem
using the appropriate tool.
- New properties Disk{fmt,spec} are added to all $gho, containing
the format used for the root disk and the xl diskspec to load it.
- lvm backed guests use a xend/xm compatible spec, everything
else uses the improved xl syntax which libvirt also supports.
We won't test non-LVM on xend.
- New properties Disk{mnt,img} are added to $gho which are not using
LVM. These contain the mount point to use (configurable via
OSSTEST_CONFIG and runvars) and the full path (including mount
point) to the image itself.
- When starting or stopping a guest we arrange for the filesystem to
be (u)mounted.
- The prepearation when starting a guest copes gracefully with
the disk already being prepared.
- Hooks are called from guest_create() and guest_destroy() to
manipulate the disk as needed.
Using standalong-generate-dump-flight-runvars a representative set of
runvars is:
+test-amd64-amd64-xl-qcow2 all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
+test-amd64-amd64-xl-qcow2 arch amd64
+test-amd64-amd64-xl-qcow2 buildjob build-amd64
+test-amd64-amd64-xl-qcow2 debian_arch amd64
+test-amd64-amd64-xl-qcow2 debian_bootloader pygrub
+test-amd64-amd64-xl-qcow2 debian_diskfmt qcow2
+test-amd64-amd64-xl-qcow2 debian_kernkind pvops
+test-amd64-amd64-xl-qcow2 debian_method netboot
+test-amd64-amd64-xl-qcow2 debian_suite wheezy
+test-amd64-amd64-xl-qcow2 kernbuildjob build-amd64-pvops
+test-amd64-amd64-xl-qcow2 kernkind pvops
+test-amd64-amd64-xl-qcow2 toolstack xl
+test-amd64-amd64-xl-qcow2 xenbuildjob build-amd64
Compared to test-amd64-amd64-pygrub (which is the most similar job) and
normalising the test name the difference is:
test-amd64-amd64-SUFFIX all_hostflags arch-amd64,arch-xen-amd64,suite-wheezy,purpose-test
test-amd64-amd64-SUFFIX arch amd64
test-amd64-amd64-SUFFIX buildjob build-amd64
test-amd64-amd64-SUFFIX debian_arch amd64
test-amd64-amd64-SUFFIX debian_bootloader pygrub
+test-amd64-amd64-SUFFIX debian_diskfmt qcow2
+test-amd64-amd64-SUFFIX debian_kernkind pvops
test-amd64-amd64-SUFFIX debian_method netboot
test-amd64-amd64-SUFFIX debian_suite wheezy
test-amd64-amd64-SUFFIX kernbuildjob build-amd64-pvops
test-amd64-amd64-SUFFIX kernkind pvops
test-amd64-amd64-SUFFIX toolstack xl
test-amd64-amd64-SUFFIX xenbuildjob build-amd64
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 8 Jul 2015 12:30:58 +0000 (13:30 +0100)]
Debian: Handle lack of bootloader support in d-i on ARM.
Debian doesn't currently know what bootloader to install in a Xen
guest on ARM. We install pv-grub-menu above which actually does what
we need, but the installer doesn't treat that as a "bootloader".
Most ARM platforms end up installing a u-boot boot.scr, based on a
platform whitelist. This doesn't seem appropriate for us. Grub is not
available for arm32. For arm64 we will eventually end up with in-guest
UEFI and therefore grub-efi and things will work normally. I'm not
sure what the answer is going to be for arm32.
This patch enables the workaround for Wheezy, Jessie and Sid,
post-Jessie should be enabled as we add them. (Pre-wheezy does not
support running as a Xen guest on ARM so we don't test them at all).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 8 Jul 2015 12:30:57 +0000 (13:30 +0100)]
distros: Run one suite per day on a weekly basis
Once a week should be sufficient for these tests. Perhaps in the
future we will want to increase the frequency for the suites under
active development (testing, unstable)
For now run this on the Citrix Cambridge instance until the XenProject
instance has sufficient capacity.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 8 Jul 2015 12:30:55 +0000 (13:30 +0100)]
distros: add branch infrastructure
Since the distro nightlies are not version controlled we cannot use
the usual mechanisms for detecting regressions. Special case things
appropriately. We use an OLD_REVISION of "flight-NNN" to signify that
the old revision is another flight and not a tree revision.
A grep over $NEW_REVISION needed adjusting since NEW_REVISION is empty
in this mode, leading to "grep <filename>" which hangs waiting for
stdin.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 8 Jul 2015 12:30:54 +0000 (13:30 +0100)]
Test pygrub and pvgrub on the regular flights
Since we now have the ability to test these drop one of each of
pygrub, pvgrub-32 and pvgrub-64 into the standard flights. Omitting
the {Guest}_diver runvar causes ts-debian-di-install to use the d-i
images in the location configured via TftpDiVersion, so they are
Version Controlled along with the d-i version used for the host.
Ian Campbell [Wed, 8 Jul 2015 12:30:53 +0000 (13:30 +0100)]
distros: Support pvgrub for Wheezy too.
This requires us to install pv-grub-menu from backports, which we do
using a late_command.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
[ ijc -- dropped unnecessary quoting of comment in heredoc ]
Ian Campbell [Wed, 8 Jul 2015 12:30:51 +0000 (13:30 +0100)]
distros: add support for installing Debian PV guests via d-i, flight and jobs
This patch introduces ts-debian-di-install which can install Debian
from a netboot (PXE) debian installer image. By default it installs
from the d-i image used by osstest (using the special Xen PV guest
enabled flavour where necessary) but it can also fetch the kernel and
ramdisk from URLs specified in runvars. The resulting guests boot the
distro kernel using pygrub (pvgrub will follow).
The distros flights differ substantially from the existing flights.
Introduce make-distros-flight using the functionality previously
refactored into mfi-common. The new flight tests all versions of
Debian from Squeeze onward as an amd64, i386 and armhf guests (armhf
from Jessie onwards only) using the usual smoke tests.
Test names are suffixed -pygrub pending the addition of pvgrub
variants in a future commit.
Add the new cases to sg-run-job
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 16 Jul 2015 14:53:43 +0000 (15:53 +0100)]
cri-bisect: Provide --html-dir= option introducer
Since 0caf39e6 "Reporting: Make HTML reports for bisection flights too"
we have been running
./sg-report-flight /home/logs/logs/59620/ 59620
which is no good.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Mon, 13 Jul 2015 08:13:06 +0000 (09:13 +0100)]
mg-list-all-branches: Sort branches according to any embedded version
Many of our branches include a version number, this change results in
e.g. linux-3.0 < linux-3.4 < linux-3.10 rather than linux-3.0 <
linux-3.10 < linux-3.4, which is more natural for uses such as
./mg-all-branch-statuses.
Ian Campbell [Mon, 13 Jul 2015 08:12:29 +0000 (09:12 +0100)]
INSTALL.production: Start writing some stuff down
I wrote this ages ago while migrating the Cambridge instance to a new
VM and then forgot about it. It's incomplete and I'm not sure where I
was planning to go next, but better than nothing maybe?
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 16 Jul 2015 10:29:00 +0000 (11:29 +0100)]
ts-windows-install: Bump RAM from 512M to 1536M
Sadly we lack a proper infrastructure for handing guest memory size
and fitting it into the host. We don't currently even have in our
database the amount of RAM the host has.
The recommended minimum for Windows is 2Gby. But we have (in the
Cambridge instance) some hosts which only have 4Gby. We allow 512 for
dom0. We need to leave some spare for Xen and other overheads. So
set this to 1536 (1.5Gby) which should fit even during localhost
migration.
Later we may want to do something more sophisticated, like adjusting
this to 2048 if the host has enough, or only running the test on
bigger hosts, or something.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Wei Liu <wei.liu2@citrix.com> CC: Andrew Cooper <andrew.cooper3@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: 1536 not 1700.
Mention migration in the commit message.
Ian Campbell [Thu, 9 Jul 2015 14:30:36 +0000 (15:30 +0100)]
mg-all-branch-statuses: Show how up to date each branch is
Using report_find_push_age_info allows us to provide counts of
attempts since the last baseline on current tip as well as the first
attempt of each of those.
Since everything serialises on the repo lock I didn't bother trying to
parallelise anything.
It's a little terse to keep it in 80 chars.
cron infrastructure is added to run this in a new
$HOME/for-maintjobs.git on a daily basis and push the output to the
location given in the Publish config variable, if it is set (it is set
in production but not for cambridge).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Thu, 9 Jul 2015 08:45:29 +0000 (09:45 +0100)]
Allow rumpkernel xenstorels.test failures for all but rumpuserxen branch
This test has a long standing heisenbug which even doing 150
iterations is not reliable at detecting. It is unlikely to be fixed
until the long-standing rumpkernel build rework is completed.
So lets mark it as an allowable failure for now, except in the
rumpuserxen branch tests themselves.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 3 Jul 2015 18:30:45 +0000 (19:30 +0100)]
Reporting: Make HTML reports for bisection flights too
Provide an appropriate --html-dir option to sg-report-flight.
(Bisection flights still do not run sg-report-host-history and
sg-report-job-history, which are fairly expensive.)
Make references to jobs from sg-report-job-history and
sg-report-host-history go to the /info.html page, not the webserver
directory listing. (sg-report-flight's references already do.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 1 Jul 2015 14:49:35 +0000 (15:49 +0100)]
Email reports: Sort out recipients and Reply-to
No longer send reports, or copies, to named individuals. Instead,
send all output to 1. appropriate development mailing lists 2. the new
osstest-admin@xenproject administrator alias, and 3. Bcc the new
osstest-output list.
Bisection progress emails go only to osstest-output. In this case we
provide a Reply-To.
After this patch goes live, people interested in bisection progress
emails will find them in osstest-output. (There are a lot of these.)
Such people can also subscribe to osstest-output and get copies of
only the main test reports and then use recipient-side filtering to
see the ones they find interesting.
(Configurations for `adhoc' and `play' runs remain unchanged and still
have a tendency to refer to my personal address @citrix.)
Deployment notes:
Keir and Stefano will no longer receive automatic CCs of certain
reports, and should subscribe to the osstest-output list, with
appropriate filtering, if they care.
The new osstest-output list has (I think) been configured to accept
the BCCs.
My personal chiark mail-to-news gateway has already been subscribed to
osstest-output.
One substantial change is that the osstest admin now gets copies of
each final bisection failure.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> CC: keir@xen.org CC: stefano.stabellini@eu.citrix.com CC: Lars Kurth <lars.kurth@citrix.com>
Ian Campbell [Thu, 2 Jul 2015 12:44:32 +0000 (13:44 +0100)]
ap-fetch-version: Arrange for osstest merges from upstream to be stable
If a downstream osstest instance has nothing to test it its local
pretest then it will attempt to merge from the upstream instance. If
this fails then it will try again and again generating a new merge
commit each time, even if upstream has not moved.
It is desirable that these merges instead be stable i.e. the same if
the inputs have not changed. This is good for potential bisection
attempts, history reporting/mining as well as just being sensible.
Here we arrange for this by recording the last merge "epoch" (being
the first merge of the current input branches) in a new branch
"merge-epoch" in the local testing.git and comparing our fresh merge
against it.
If the tree and parents are the same then the merge is effectively
identical (it may/will differ in the date) and we reuse the epoch
merge.
If they new merge does not match then something has changed (i.e.
upstream has moved on) and so we take the new merge and establish a
new epoch.
Add a couple of ">&2" to some fetches to prevent anything other than
the desired SHA1 leaking onto stdout.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 3 Jul 2015 11:22:52 +0000 (12:22 +0100)]
ts-logs-capture: Explicitly set xenctx capture filename
Now that the command contains PATH, the autogenerated filename is very
ugly.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch in this version of the series.
Ian Jackson [Thu, 2 Jul 2015 17:05:25 +0000 (18:05 +0100)]
ts-logs-capture: Run xenctx earlier
Break fetch_xenctx_guest out into its own function, and run it before
serial_fetch_logs.
This is relevant because serial_fetch_logs sends the Xen debug keys,
which might dislodge a stuck guest - and, if it does, we would like to
have captured the stuck state with xenctx, rather than the unstuck
one.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 3 Jul 2015 10:06:40 +0000 (11:06 +0100)]
sg-report-flight: Fix --include (broken by 3d56a191)
In 3d56a191 "sg-report-flight: Make bodyprint[f] defer the output" it
became wrong for anything in the main printout part of
sg-report-flight to write directly to STDOUT. However, the two (very
similar) bits of code for handling the two kinds of --include still
used cat.
The result is emails looking like this:
From: osstest service owner <osstest-admin@xenproject.org>
To: <xen-devel@lists.xensource.com>, <rumpkernel-builds@freelists.org> CC: <ian.jackson@eu.citrix.com>
Subject: [rumpuserxen test] People who touched revisions under test:
Alice Bob <alice@example.com>
Ian Jackson <Ian.Jackson@eu.citrix.com>
Carol David <carol@example.com>
Date: Fri, 3 Jul 2015 06:50:53 +0000
flight 59037 rumpuserxen real [real]
Fix this by replacing those two near-identical bits of code with calls
to a single function which slurps the required include file into the
body text accumulation variable.
(There is no need to provide a facility for header includes, since
there is no need for a caller to pass headers through
sg-report-flight: they can simply be written into the output file
before running sg-report-flight, as indeed the string `Subject: ' is.
Despite the fact that --include-begin is used only by cr-daily-branch
which passes it a file `heading', the text is intended for the start
of the body, not the email headers.)
(b) The output after this patch differs from the output produced by 1f0c168d (the previous osstest production commit) only in addition of
the `Last test of basis' and `Testing same since' information, in the
appropriate place.
Deployment note: Because this patch changes only sg-report-flight, it
is safe to deploy under the feet of a running flight. I intend to do
this for all the production branches in the colo.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Dario Faggioli [Tue, 30 Jun 2015 17:36:59 +0000 (19:36 +0200)]
make-flight etc.: stop testing SEDF at all
SEDF has been broken and unmaintained at least until Xen 4.2, and most
likely even before! Tests are failing without anyonce caring, and yet
we're keeping using test resources for them.
Let's stop doing this!
Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 30 Jun 2015 13:31:28 +0000 (14:31 +0100)]
Emails: Provide X-Osstest-Failures
Specifically:
* $cat now contains SORTLETTER BLOCKSTATUS DESCRIPTION...
* Rewrap the lines setting $cat
* Add $notsucceeds{$cat}[][2] containing one line for the new header
* Generate the new header
* Document the new header
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Tue, 30 Jun 2015 13:29:21 +0000 (14:29 +0100)]
Emails: Provide X-Osstest-Versions-This and That headers
Specifically:
* Provide headerprint() in sg-report-flight
* Use it to add X-Osstest-Versions headers based on %specver
* Cause cr-daily-branch to always enable the feature
* Document this in README.email
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Document This and That
Ian Jackson [Tue, 30 Jun 2015 13:17:05 +0000 (14:17 +0100)]
Emails: Provide README.email and link to it
Also link to README, from the email footer.
I have tested that the link to README works right now. The link to
README.email does not yet, of course, but it will start to work when
osstest pushes this change to production.
CC: Andrew Cooper <andrew.cooper3@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Tue, 30 Jun 2015 10:08:37 +0000 (11:08 +0100)]
sg-report-flight: Use `bodyprint' and `bodyprintf'
This will allows us to divert the output so that we can (optionally)
produce email headers too. We move the error checking into the
function, obviously.
For now, this is a simple replacement with no functional change,
except that some call sites now have error checking which previously
did not.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 29 Jun 2015 17:06:11 +0000 (18:06 +0100)]
Reporting: sg-report-flight: Put overall pushgate stats in email
This adds something like this (would have appeared in 37638):
Last test of basis 37629 2015-06-24 21:36:10 Z 4 days
Failing since 37630 2015-06-26 10:35:44 Z 3 days 9 attempts
Testing same since 37635 2015-06-28 03:05:56 Z 1 days 4 attempts
or this (would have appeared in 37629):
Last test of basis 37617 2015-06-20 04:06:01 Z 9 days
Testing same since 37628 2015-06-24 13:35:23 Z 5 days 2 attempts
(Both examples from the Cambridge instance.)
This is added just after the versions are reported.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Swap order of `Failing since' and `Testing same since' output lines
v3: Actually return when no pushgate summary wanted.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Fix bug resulting in ERROR: missing FROM-clause entry for table "r"
Ian Jackson [Mon, 29 Jun 2015 15:46:36 +0000 (16:46 +0100)]
Reporting: In history tables, show multiple failure steps
In report_run_getinfo, used by sg-report-job-history and
sg-report-host-history, make provision for reporting multiple failed
steps.
The failed steps are put side by side (in a single table cell). The
cell background is set by the job status, but the individual steps may
vary their colour, as seen in this example:
http://xenbits.xen.org/people/iwj/2015/linux-next.html
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Remove unnecessary space in ColourAttr
Ian Jackson [Mon, 29 Jun 2015 15:23:08 +0000 (16:23 +0100)]
Reporting: Break out report_blessingscond
The returned SQL is a self-contained expression, and does not require
additional bind parameters. To spot SQL quoting problems, we die if
a blessing is not reasonable.
The use sites of the $blessingcond in sg-report-flight are adjusted to
no longer pass @blessings to execute.
No overall functional change with reasonable blessings.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 29 Jun 2015 14:55:13 +0000 (15:55 +0100)]
Reporting: report_run_getinfo produces Content
Previously we produced Summary, and the call sites were confused about
whether it had already been html-escaped. Now we produce something
which is explicitly already html.
No functional change except that some sg-report-host-history output is
escaped as it ought to have been (but this is only relevant if the
database contains strange things which none of ours do).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 29 Jun 2015 14:51:22 +0000 (15:51 +0100)]
Reporting: report_run_getinfo produces ColourAttr
This includes the bgcolour attribute name and the quotes. This will
make it possible for this function to sometimes not set a background
for the whole table cell in the future (which will be part of putting
multiple step results in each cell).
No functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Remove an unneeded space in ColourAttr
Ian Jackson [Mon, 29 Jun 2015 15:37:14 +0000 (16:37 +0100)]
Reporting: sg-report-host-history: Support --max-flight
We can't use report_blessingscond because we handle blessings
differently in sg-report-host-history (since we want the history of
even adhoc etc. flights).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 26 Jun 2015 18:21:34 +0000 (19:21 +0100)]
Reporting: sg-report-flight: Fix undefined value in $worstrow[]
The fallback entry in the worstrow search, used when a job has not
been run at all, was wrong. Also, fix the doc comment which induced
the mistake, and add a comment about the contents of @worst.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Added the comment about @worst.
Ian Jackson [Tue, 30 Jun 2015 09:22:25 +0000 (10:22 +0100)]
ap-common: Abolish linux-3.0 special case
This is identical to the effect of the linux-[3-9] etc. glob.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch in this version of the series.
Ian Jackson [Mon, 29 Jun 2015 10:00:48 +0000 (11:00 +0100)]
Test Linux 4.1
The kernel.org stable git branch layout is the same for 4.1 as for
3.x, so simply extend the pattern, and add the branch to BRANCHES.
We move the linux-stable.git pattern to the end, and extend it to
linux-[3-9].[0-9]*, to provide a bit more future-proofing. (There is
no particular reason other than shell glob convenience to stop at
linux-9.x.)
DEPLOYMENT NOTE: After this patch is acked, but just before it is
pushed to pretest, run (this version of)
./mg-branch-setup linux-4.1 bisect b953c0d234bc72e8489d3bf51a276c5c4ec85345
to create the working trees on the osstest VM and push the base of
v4.1 to the tested output branch.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Extend glob pattern in ap-common and move to the end.
Ian Campbell [Mon, 15 Jun 2015 11:37:44 +0000 (12:37 +0100)]
Debian: Arrange to be able to chainload a xen.efi from grub2
Xen cannot (currently) be booted directly via the usual multiboot
path on EFI systems of any arch. Instead it is necessary to either
launch xen.efi direct from the UEFI shell or to chainload it from
grub. In both cases the Xen command line as well as what would
normally be the multiboot modules (kernel+command line, XSM policy,
initrd) must be configured in a Xen configuration file.
Here we add a new overlay/etc/grub.d/15_osstest_uefi grub script which
arranges that if a xen.efi is found in the EFI System Partition (as
arrange by a previous patch) a suitable entry is created in grub.cfg
as well.
When parsing the grub.cfg look for such an entry into addition to the
regular/multiboot one and when necessary write the configuration file
based on the regular entry and return the chainload one such that it
gets booted.
This is currently enabled only for Jessie ARM64 systems.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Use a separate grub script snippet, there's nothing we need in
20_linux_xen
v3: Rewrap and reindent to avoid long lines
Ian Campbell [Mon, 15 Jun 2015 10:42:18 +0000 (11:42 +0100)]
Collect xen.efi into xendist and install in appropriate place
Previously these binaries would have been included in the regular
("tools") dist file, whereas they really belong in the xen one.
Install into /boot/efi/EFI/osstest ready for use when chainloading.
Note that /boot/efi is (or should be) a VFAT filesystem. So a bit of
care is needed WRT symlinks etc. This is also what prevents us from
just including /boot/efi/EFI/osstest/xen.efi in the dist tarball since
untarring over a VFAT needs a little care WRT the case of directories
etc.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Thu, 28 May 2015 11:44:34 +0000 (12:44 +0100)]
Debian: grub2: Use GRUB_CMDLINE_LINUX_XEN_REPLACE(_DEFAULT)
This overrides GRUB_CMDLINE_LINUX(_DEFAULT) which we were previously
editing but only for the Xen entries, meaning that we don't switch to
console=hvc0 for the native cases (i.e. don't break them).
We do however want to edit GRUB_CMDLINE_LINUX(_DEFAULT) to remove
"quiet" if present, since it is useful to people, especially those
using standalone mode as a provisioning tool, wanting to e.g. compare
a boot under Xen with the native case.
This has been supported since Wheezy but in any case we supply our own
20_linux_xen grub generator based on Wheezy which supports this.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 24 Jun 2015 08:23:16 +0000 (09:23 +0100)]
Debian: Do not add Debian to UEFI boot order on install
By default Debian inserts itself before any existing entries,
including the PXE one, meaning we otherwise cannot remotely regroove
the box.
Add a hook to /usr/lib/base-installer.d/ which installs a diversion of
/usr/sbin/grub-install and installs a wrapper which adds the
--no-nvram argument which prevents grub-install from adding the boot
entries.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Die on !jessie
v3: New patch replacing "Debian: Fixup UEFI boot order during install"
Ian Campbell [Thu, 21 May 2015 09:57:41 +0000 (10:57 +0100)]
ts-host-install: Set dtbs in the non-special kernel case too.
When DTB support was added for armhf it required a backports kernel
for use with Wheezy and therefore DTBs were only collected for that
special case. However as we move forward to newer kernels it should
be possible to run using the proper kernel (e.g. Jessie should work
for armhf). Therefore we will need the DTBs for those kernels too.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v3: Improved commit message
Ian Campbell [Wed, 20 May 2015 15:58:16 +0000 (16:58 +0100)]
ts-kernel-build: Additional kernel options for Mustang
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Note: ARM testing currently uses a 3.16 based linux-arm-xen kernel
(compared to 3.14 currently used as the default). In order to support
Mustang something newer is needed, I've been testing 4.1 (next LTS
kernel), 3.18 (the intervening LTS kernel) might be sufficient but I
haven't tried it. We should plan to either update linux-arm-xen or the
default kernel version at some point before we get Mustang h/w.
v3: Set CONFIG_NET_XGENE (was commented out before)
Remove setting of CONFIG_SPARSEMEM_VMEMMAP
Ian Campbell [Thu, 21 May 2015 12:58:27 +0000 (13:58 +0100)]
Enable chain loading to local disk for UEFI PXE systems.
First arrange for bootloader to be installed to removable media path,
by using a new in Jessie preseed option. Then use that to chainload a
bootloader from the disk.
The removable media path is well known (part of the UEFI spec) which
saves us having to worry about which OS is on the host (so long as the
OS can be configured to populate the removable media path)
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 27 Jan 2015 12:06:41 +0000 (12:06 +0000)]
ts-host-install: Support UEFI PXE boot using grub.efi
Handling of chainloading from the local hdd in
setup_pxeboot_local_uefi will be added in the next patch.
(Wrap DiBase entry in README too for consistency)
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Turn grub "echo todo" into a die.
v3: Reindent README (fix DiBase too for consistency)
Adjust for reordering of setup_{firmware}_pxeboot to put
{firmware} at end
Rewrap setup_grub_efi_bootcfg
Adjust for setup_pxeboot -> setup_pxeboot_di
Ian Campbell [Thu, 21 May 2015 12:46:24 +0000 (13:46 +0100)]
Refactor pxelinux configuration
The mechanism used to PXE boot can differ depending on the firmware
type. Therefore refactor into Osstest::TestSupport and key off a new
host property "firmware".
Currently supported is "bios" (the default) and "uboot", both of which
use pxelinux.cfg style files.
The default for the firmware property is "bios", hence no change for
any existing system (including those which use uboot, regardless of
whether they are now configured with the firmware proprty or not)
As part of this "setup_pxeboot" becomes "setup_pxeboot_di" since it is
somewhat d-i specific.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v3: reorder setup_${firmware}_pxeboot to setup_pxeboot_${firmware}
join $dicmd in setup_pxeboot instead of caller
Ian Campbell [Thu, 21 May 2015 11:49:55 +0000 (12:49 +0100)]
ts-host-install: split the "di" from the "host" command line
(i.e. the bit before/after the -- marker). When abstracting over
different bootloaders in a future patch this will be convenient since
it allows the code to add to either.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Thu, 21 May 2015 11:46:16 +0000 (12:46 +0100)]
ts-host-install: Split initrd out of @installcmdline
Other bootloaders handle this with an explicit separate option rather
than parsing it out of the command line as pxelinux does. Prepare for
supporting these.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 27 Jan 2015 10:07:21 +0000 (10:07 +0000)]
Introduce mg-pxe-loader-update
The story for PXE booting via UEFI (at least on arm64) is not so
straightforward as with pxelinux on x86. There seems to no good
bootloader to launch via UEFI+pxe, in fact all I could find was grub
(syslinux, and by extension pxelinux.efi, is x86 only).
Add mg-pxe-loader-update modelled on mg-debian-installer-update which
will download the necessary grub binaries and produce a grub image
which can be used to pxe boot.
grub lacks the convenient ability to search for config file based on
(substrings of) the MAC or IP address. So we arrange for the grub.cfg
in TftpGrubBase to chain load another config file from
TftpTmpDir/'$net_default_mac' where $net_default_mac is a grub
variable which is substituted at boot time.
Actually using this requires that bootp/dhcp provide a next-file so
UEFI knows what to boot (usually this would be pxelinux.0 which we
can't use here). Locally we have configured this as $name/pxe.img, so
we can use different loaders.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Just mgi-common now that mgi-debian merged therein.
v3: Use mgi-common and mgi-debian
Ian Campbell [Fri, 19 Jun 2015 10:56:15 +0000 (11:56 +0100)]
mg-*: Make Debian package fetching common in mgi-common
This pattern appears several times in mg-debian-installer-update and
I'm about to add another script which uses it.
The centralised version always does "sort -n -r | head -n" to pick out
only the newest version, which only some of the open coded sites did.
Likewise the centralised version checks that $pkgfile is non-empty,
unlike some of the open coded sites.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Fold into mgi-common instead
v3: New patch
projects / osstest.git / log