]> xenbits.xensource.com Git - people/dariof/osstest.git/log
people/dariof/osstest.git
10 years agomg-debian-installer-update: Add more modules when creating initrd overlay
Ian Campbell [Wed, 29 Oct 2014 10:41:25 +0000 (10:41 +0000)]
mg-debian-installer-update: Add more modules when creating initrd overlay

The arndale platform has a usb network device. In addition both the SATA and
NIC/USB hardware on this platform requires additional regulator (power), clk
and phy modules, which in turn require i2c support.

Add everything which is needed on this platform.

Sort the list of drivers while here.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agomg-debian-installer-update: Include any device tree blobs in the tftp dir
Ian Campbell [Wed, 29 Oct 2014 10:41:24 +0000 (10:41 +0000)]
mg-debian-installer-update: Include any device tree blobs in the tftp dir

We will need to supply these for some arm boards.

We provide both an unpacked version, which can be referenced via u-boot's
support for the "fdtdir" property in pxelinux.cfg and a tarball which is more
convenient for installing into the target filesystem on host install

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agostandalone: Provide a helper to clear the host runvar
Ian Campbell [Wed, 29 Oct 2014 10:41:23 +0000 (10:41 +0000)]
standalone: Provide a helper to clear the host runvar

This clears the host flag used for single-host test cases.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoTftpDiVersion 2014-10-20
Ian Jackson [Mon, 20 Oct 2014 12:11:17 +0000 (13:11 +0100)]
TftpDiVersion 2014-10-20

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-for-branches: Keep many more logs
Ian Jackson [Mon, 6 Oct 2014 16:05:32 +0000 (17:05 +0100)]
cr-for-branches: Keep many more logs

Keep 300 cycles of cr-for-branches.log.

Also, remove the spurious `500' from the command line, which looks
like it was there because I thought that `-n' took a number argument
(but in fact that's -c), whereas it was actually interpreted as a
filename (which never exists so savelog silently ignores it).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agobisection: Do not get bisection start points from broken flights
Ian Jackson [Mon, 6 Oct 2014 15:54:55 +0000 (16:54 +0100)]
bisection: Do not get bisection start points from broken flights

cs-bisection-step ignores any flight with broken steps, on the grounds
that the results in such a flight maybe unreliable.

Therefore, if a main flight has both broken steps and regressions, it
is wrong to take its mro as the new basis mro for bisections.  In the
worst case a newly failing job will be determined to be unbisectable
(since the only relevant failure is in a broken flight), have the
stamp file created, and then never be properly bisected.

So: in sg-report-flight, report `broken-step' for each broken step (in
the top-level flight we are reporting - broken jobs in flights we are
referring to to justify heisenbugs are OK); and, in cr-daily-branch,
do not install the new mro if such indications are present.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocat-dhcp-leases: new script
Ian Jackson [Fri, 3 Oct 2014 15:02:32 +0000 (16:02 +0100)]
cat-dhcp-leases: new script

ISC dhcpd treates /var/lib/dhcp3/leases as a log-structured database,
appending records to it and periodically rewriting it.  When it
decides to rewrite it it writes the new contents to a temporary file -
fine.

But rather than renaming the temporary file into place, over the top
of the old file, using rename(2), it first renames the old leases file
to leases~.  So briefly there is no leases file.

This means that the service used by Osstest/DhcpWatch/leases.pm when a
hostname and port is specified has to try various different files.

This is the appropriate shell script, which is most conveniently
maintained here in osstest.  It can be used by putting something like
this in inetd.conf:

5556 stream tcp nowait nobody /root/cat-dhcp-leases cat-dhcp-leases

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agosg-report-flight: Add a succcess/failure column header.
Ian Jackson [Thu, 2 Oct 2014 11:56:06 +0000 (12:56 +0100)]
sg-report-flight: Add a succcess/failure column header.

There are so many rows and columns that scrolling around trying to find the
column associated with a failure is getting pretty tricky. Finding all failing
columns is even harder.

Add a clue near the top: replicate the worst or most interesting step
result box contents and colour.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agots-rumpuserxen-build: Double the build timeout
Ian Jackson [Fri, 26 Sep 2014 15:35:18 +0000 (16:35 +0100)]
ts-rumpuserxen-build: Double the build timeout

The build is quite big and in particular configure is very iobound and
sensitive to other load on the box.  We have seen spurious timeouts.

CC: Jan Beulich <JBeulich@suse.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoMerge branch 'pretest' of ssh://osstest/export/home/osstest/testing.git/ into pretest
Ian Campbell [Tue, 23 Sep 2014 12:54:05 +0000 (13:54 +0100)]
Merge branch 'pretest' of ssh://osstest/export/home/osstest/testing.git/ into pretest

10 years agoSwitch armhf to use common default Linux.
Ian Campbell [Thu, 4 Sep 2014 14:16:39 +0000 (15:16 +0100)]
Switch armhf to use common default Linux.

Previous we had a v3.13 based branch which was used for ARM (because
v3.4 was too old). Now that the default is v3.14 we can use it.

standalone-generate-dump-flight-runvars before and after shows the
build-armhf-pvops jobs' linux_revision runvar changing from
95bfbee422b9b1cfe8c2d2e27edf17ce1cc99e04 (v3.13-63-g95bfbee) to
946de0e6b6ed49eacb03e3cddfcb1d774d6378ed (v3.14.17) for these branches:

libvirt
osstest
qemu-mainline
qemu-upstream-unstable
xen-4.3-testing
xen-4.4-testing
xen-unstable

This is as expected.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoSwitch default Linux branch from 3.4 to 3.13
Ian Campbell [Thu, 4 Sep 2014 14:02:28 +0000 (15:02 +0100)]
Switch default Linux branch from 3.4 to 3.13

standalone-generate-dump-flight-runvars before and after shows revision_linux
changing from 9f089cfb4be27efe7c1cd1d6261da02496820441 (v3.4.103) to
946de0e6b6ed49eacb03e3cddfcb1d774d6378ed (v3.14.17) for the
build-{amd64,i386}-pvops jobs these branches:

osstest
rumpuserxen
xen-4.0-testing
xen-4.1-testing
xen-4.2-testing
xen-4.3-testing
xen-4.4-testing
xen-unstable
qemu-mainline
qemu-upstream-4.3-testing
qemu-upstream-4.4-testing
qemu-upstream-unstable
seabios
libvirt
qemu-upstream-4.2-testing

build-armhf-pvops and build-*-oldkern jobs as well as the linux-* branches are
unaffected. Which is as expected.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agomg-hosts mkpxedir: honour $OSSTEST_SUDO
Ian Jackson [Tue, 19 Aug 2014 10:33:15 +0000 (11:33 +0100)]
mg-hosts mkpxedir: honour $OSSTEST_SUDO

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoFix mg-hosts mkpxedir
Ian Jackson [Tue, 19 Aug 2014 10:31:17 +0000 (11:31 +0100)]
Fix mg-hosts mkpxedir

 * Have selecthost no longer demand that $job is set; if it is not,
   simply do not set Suite (which of course ultimately comes from
   runvars and hence the job).

 * Add missing use Osstest::TestSupport.

 * Mention the need to run it in README.dev.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agosg-report-flight: New --branches-also option
Ian Jackson [Thu, 31 Jul 2014 15:26:55 +0000 (16:26 +0100)]
sg-report-flight: New --branches-also option

This allows easy comparison of relevant flights (specified by tree
versions) from one branch with relevant flights from another.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agorump kernel tests: Get upstream from rumprun-xen
Ian Jackson [Thu, 31 Jul 2014 14:42:56 +0000 (15:42 +0100)]
rump kernel tests: Get upstream from rumprun-xen

Replace https://github.com/rumpkernel/rumpuser-xen
with https://github.com/rumpkernel/rumprun-xen
as the source for the pre-test version.  The repo has been renamed
upstream.

We do not change, at this time:

 * The name of the post-test output repo on xenbits,
     git://xenbits.xen.org/rumpuser-xen.git#xen-tested-master

Changing those would be a bit fiddly.

And we don't change the following, which contain `rumpuserxen':

 * osstest `branch', `recipe', and `jobs' which are used to correlate
   and specify aspects of the different production test runs of
   different versions of the code.  The branch name appears in the
   Subject lines of email reports, and the job names in the body (and
   as column names in the html test report matrix).

 * The osstest `tree' names (which are used to specify and correlate
   the different versions and urls of the `same' codebase in the
   various tests).  Also, we leave unchanged the corresponding
   osstest tree plumbing environment variable names.

 * The osstest build output identifiers from ts-rumpuser-xen-build
   including the runvar name and the dist part name.  (These are used
   to communicate between the build and test jobs.)

Changing those would break cross-change comparison of test results and
osstest's automatic bisection across the change.

We also don't change:

  * The names of the ts-rumpuserxen-* test scripts.

While this could be changed by renaming the files and editing
sg-run-job to match, it doesn't seem sensible to rename these while
leaving everything else unchanged.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoTestSupport: Set DISTCC_FALLBACK=0 when using distcc
Ian Campbell [Mon, 28 Jul 2014 13:01:20 +0000 (14:01 +0100)]
TestSupport: Set DISTCC_FALLBACK=0 when using distcc

This will show up issues with the distcc infrastructure immediately
instead of limping on with reduced performance.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agocr-for-branches: add linux-3.14
Ian Jackson [Fri, 25 Jul 2014 16:23:52 +0000 (17:23 +0100)]
cr-for-branches: add linux-3.14

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoREADME.dev: Slightly tart up the new branch instructions
Ian Jackson [Fri, 25 Jul 2014 16:23:40 +0000 (17:23 +0100)]
README.dev: Slightly tart up the new branch instructions

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomake-flight: Enable qemu tests for 4.5 onwards
Ian Campbell [Wed, 16 Jul 2014 11:45:05 +0000 (12:45 +0100)]
make-flight: Enable qemu tests for 4.5 onwards

Stefano is about to enable qemu support on ARM, we should test it.

standalone-generate-dump-flight-runvars verifies that the affect of this change
is to add armhf jobs/runvars to qemu-mainline and qemu-upstream-unstable as
desired.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Stefano.Stabellini@eu.citrix.com
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agostandalone-generate-dump-flight-runvars: allow 26 chars for branch name
Ian Campbell [Wed, 16 Jul 2014 11:44:46 +0000 (12:44 +0100)]
standalone-generate-dump-flight-runvars: allow 26 chars for branch name

qemu-upstream-X.Y-testing is 25 characters. Also add a space.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agorump kernel tests: Run xenstore-ls demo
Ian Jackson [Mon, 14 Jul 2014 12:08:58 +0000 (13:08 +0100)]
rump kernel tests: Run xenstore-ls demo

We can use the newly-generalised ts-rumpuserxen-demo-setup to set up
the config file etc. to run xenstore-ls from the xen.git build.

We have a bespoke script to actually execute it: we extract the actual
program output from the guest console log, and compare the results to
those we expect (based on running xenstore-ls in dom0).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Fix copyright date in new script.

10 years agots-guest-destroy-hard: New script
Ian Jackson [Tue, 15 Jul 2014 14:18:00 +0000 (15:18 +0100)]
ts-guest-destroy-hard: New script

This just calls xl destroy; it doesn't check that the guest is
running, or on the network, or anything.  (It also doesn't check that
the guest is down afterwards.)

We are going to want this to be able to destroy the rumpuserxen
xenstorels domain, which is short-lived (and will normally have
terminated by the time we want to destroy it).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix copyright date in new script.

10 years agorump kernel tests: Specify cmdline via runvar
Ian Jackson [Mon, 14 Jul 2014 11:58:57 +0000 (12:58 +0100)]
rump kernel tests: Specify cmdline via runvar

The runvar <gn>_cmdline is used for the rumpuserxen guest command
line.  Previously "3" (a binary flags word meaningful to the test/demo
program included in rumpuser-xen.git) was hardcoded.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agorump kernel tests: Specify kernel via runvar
Ian Jackson [Tue, 15 Jul 2014 14:40:08 +0000 (15:40 +0100)]
rump kernel tests: Specify kernel via runvar

The runvar <gn>_builtimage is set to <distpart>:<distpath>.
Previously we hardcoded `/usr/local/lib/xen/rump-kernel' in
`rumpuserxen'.

The ffs filesystem images, and the demo config template, still come
from a single particular build, which we find from the
`guests_rumpuserxenbuildjob' runvar.  This runvar is then doing double
duty as the build job for the ffs filesystem images and example
configs, and the `default' build job for guest kernels.

The subpath /usr/local/lib/xen/rump-kernel is duplicated.  It
now appears:
 1. To find the demo guest's kernel: in make-flight, setting a runvar,
    with the actual rump-kernel kernel image leafname appended.
 2. For the filesystem images and config file: hardcoded in
    ts-rumpuserxen-demo-setup.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix demo application kernel image filename.
    Disentangle wording of last para of commit message.

10 years agorump kernel tests: Use guest-specific fs images location
Ian Jackson [Tue, 15 Jul 2014 14:38:11 +0000 (15:38 +0100)]
rump kernel tests: Use guest-specific fs images location

We are going to want to have multiple different guests in the same
job.  Don't use the same locations for all of their fs images.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agorump kernel tests: Do a xen.git tools build
Ian Jackson [Mon, 14 Jul 2014 12:04:35 +0000 (13:04 +0100)]
rump kernel tests: Do a xen.git tools build

Having laid the groundwork, we can now:

* In make-flight, for the rumpuserxen builds, set the runvars that
  specify which xen.git bits etc. to build.  (We pass qemuu now,
  because we intend that eventually we will want to build that too.)

* In ts-rumpuserxen-build, record the locations of the app-tools
  configure and make wrappers in the runvars which will be honoured by
  ts-xen-build.  Thus ts-xen-build in rumpuserxen build jobs will use
  these app-tools and do a rump cross build.

* Add a ts-xen-build step to the build-rumpuserxen recipe, passing
  `tools' as the target to build.

ts-rumpuserxen-build does not generate a formal set of build outputs.
Instead, the ts-xen-build step references the files generated by
ts-rumpuserxen-build directly (and is told where to find them via the
new command prefix runvars).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Improve commit message.

10 years agots-xen-build: Honour cmdprefix_configure, cmdprefix_make
Ian Jackson [Mon, 14 Jul 2014 12:03:16 +0000 (13:03 +0100)]
ts-xen-build: Honour cmdprefix_configure, cmdprefix_make

These runvars can be set to specify a string (containing shell code)
to be prefixed to the executions of ./configure and make.

If not set, the behaviour is as as before.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agots-xen-build: Allow specification of make targets
Ian Jackson [Mon, 14 Jul 2014 12:01:37 +0000 (13:01 +0100)]
ts-xen-build: Allow specification of make targets

This allows sg-run-job (or a manual caller of ts-xen-build) to specify
which target(s) to pass to `make'.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agomake-flight: rumpuserxen tests: use same buildjob for all guests
Ian Jackson [Mon, 14 Jul 2014 11:52:28 +0000 (12:52 +0100)]
make-flight: rumpuserxen tests: use same buildjob for all guests

Change the name of the runvar for the build job to be
guests_rumpuserxenbuildjob, which is the default (fallback) for
guest_var.

(We are going to introduce another step which uses another guest but
wants the same build job.)

No resulting change in the system's overall behaviour.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix stray leftover in commit message.

10 years agobuilding: Delete when cloning, not when preparing $builddir
Ian Jackson [Mon, 14 Jul 2014 11:37:43 +0000 (12:37 +0100)]
building: Delete when cloning, not when preparing $builddir

Make prepbuilddirs no longer delete $builddir; instead:
 - have prepbuilddirs delete *-stamp files and dist/;
 - have build_clone delete the specific source directories;
 - any individual ts-*-build scripts that use other things
   would have to delete them (but searching for $builddir
   reveals none).

This makes it possible for later steps within a build job to look at
the build trees from previous steps.  This is helpful if for any
reason it isn't easy or desirable to package up a formal set of build
outputs for reuse.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Provide motivation in the commit message.

10 years agosg-run-job: Support toggling `adding' by repeating `+'
Ian Jackson [Mon, 14 Jul 2014 12:07:30 +0000 (13:07 +0100)]
sg-run-job: Support toggling `adding' by repeating `+'

Specifying `+' as an argument to spawn-ts suppresses the automatic
addition of subsequent arguments to the testid.  Make it possible to
toggle this by repeating `+'.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoBuildSupport: make selectbuildhost modify @ARGV
Ian Jackson [Fri, 11 Jul 2014 14:49:10 +0000 (15:49 +0100)]
BuildSupport: make selectbuildhost modify @ARGV

We pass it \@ARGV everywhere now, and it eats the argument (if it
finds one).  This allows build scripts to take extra arguments.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoselectguest: log a message
Ian Jackson [Tue, 15 Jul 2014 14:15:02 +0000 (15:15 +0100)]
selectguest: log a message

Print the guest's name and the host it's on; this can help with
interpreting the logs.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agots-rumpuserxen-build: Cope with new layout, disabling bodges
Ian Jackson [Fri, 11 Jul 2014 14:27:11 +0000 (15:27 +0100)]
ts-rumpuserxen-build: Cope with new layout, disabling bodges

The rumpsrc psuedo-submodule has been integrated with the nblibs
submodule and made into a proper submodule.

Also, in the intervening time, rumpuserxen/Config.mk now uses ?= when
setting XEN_HEADERS.

So if we detect that we have a new rumpuser-xen, we can disable:
 - ad-hoc arrangements for rumpsrc
 - grievous-bodge
 - patching Config.mk

We consider a rumpuserxen old iff it has nblibs as a submodule and
lacks rumpsrc as a submodule.

We have to have a different name for the rumpsrc submodule (rather
than using rumpsrc), because in the new arrangements it needs to be
fetched from the location specified in rumpuserxen's .gitmodules
rather than our own configured location for the old rumpsrc.  (And the
local name is in the runvars etc.)

The old runvars etc. for rumpsrc need to stay around to cope with old
rumpuserxen, for now, but are going to be obsolete.  Leave some
comments about this in relevant places.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoBuildSupport: provide submodule_have
Ian Jackson [Fri, 11 Jul 2014 14:13:03 +0000 (15:13 +0100)]
BuildSupport: provide submodule_have

Convenience function.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoadhoc-revtuple-generator: Explicitly specify refs to fetch
Ian Jackson [Tue, 8 Jul 2014 10:27:59 +0000 (11:27 +0100)]
adhoc-revtuple-generator: Explicitly specify refs to fetch

Some of the git trees in $c{Repos} may have been created (eg by
earlier versions of git or earlier versions of osstest) without the
usual
  remote.origin.fetch = +refs/heads/*:refs/remotes/origin/*
git config setting.

Pass this refspec explicitly, instead of relying on it being set as
the default.

Amongst other things, this will prevent a possible error during
bisection
  git fetch -p origin
  [stuff]
  fatal: bad object 3e3d5f6146c3f684f4b7bd71feffc31e1c668754
which is currently appearing in our cron mail.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoexecutive-postgresql-schema: Schema dump
Ian Jackson [Mon, 30 Jun 2014 10:13:37 +0000 (11:13 +0100)]
executive-postgresql-schema: Schema dump

We really ought to have a proper schema script.  But for now, include
the output of pg_dump --schema-only on the osstestdb, for the benefit
of anyone trying to set up a hosted (rather than standalone) instance
(or just trying to see how the code works).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New patch in v2 of the series.

10 years agomake-flight: Test rump kernels only when needed
Ian Jackson [Fri, 27 Jun 2014 10:20:19 +0000 (11:20 +0100)]
make-flight: Test rump kernels only when needed

Introduce a new subroutine branch_wants_rumpkernel_tests and call it
before making the rumpuserxen build and test jobs.

The default is not to include a rumpuserxen test.  We include it for
rumpuserxen itself, xen, linux (in case there is a frontend/backend
incompatibility), and osstest.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomake-flight: Break out do_rumpkernel_tests
Ian Jackson [Fri, 27 Jun 2014 10:19:47 +0000 (11:19 +0100)]
make-flight: Break out do_rumpkernel_tests

No functional change in this patch.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agostandalone-generate-dump-flight-runvars: New utility
Ian Jackson [Fri, 27 Jun 2014 14:40:57 +0000 (15:40 +0100)]
standalone-generate-dump-flight-runvars: New utility

This is helpful for showing the effect of changes to make-flight,
cr-daily-branch, et al.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Do not set DAILY_BRANCH_POSTMAKE_HOOK, since standalone now does
    that.

10 years agomg-list-all-branches: New script.
Ian Jackson [Fri, 27 Jun 2014 11:45:11 +0000 (12:45 +0100)]
mg-list-all-branches: New script.

The approach here is a little unpleasant but it has the virtue of
getting the ones out of crontab too, and of not depending on the
production database (which will in any case contain obsolete flights).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agostandalone-config-example: Provide a setting of Repos
Ian Jackson [Fri, 27 Jun 2014 15:51:46 +0000 (16:51 +0100)]
standalone-config-example: Provide a setting of Repos

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agostandalone: Use DAILY_BRANCH_POSTMAKE_HOOK
Ian Jackson [Mon, 30 Jun 2014 10:12:08 +0000 (11:12 +0100)]
standalone: Use DAILY_BRANCH_POSTMAKE_HOOK

DAILY_BRANCH_PREEXEC_HOOK is after we have made half the prospective
email, but this is not useful for standalone users.

Reported-by: Ian Campbell <Ian.Campbell@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New patch in v2 of the series.

10 years agostandalone: Honour OSSTEST_CONFIG
Ian Jackson [Fri, 27 Jun 2014 15:51:27 +0000 (16:51 +0100)]
standalone: Honour OSSTEST_CONFIG

If OSSTEST_CONFIG is set in the environment, do not default it
to ~/.xen-osstest/config.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agosubmodulefixup: Tell git cache [fetch=try]
Ian Jackson [Fri, 27 Jun 2014 10:42:31 +0000 (11:42 +0100)]
submodulefixup: Tell git cache [fetch=try]

Make git_massage_url take an optional %xopts list, currently with
only GitFetchBestEffort as a possible key.

Use that in submodulefixup, unconditionally.  Often the submodule
update will work fine with an older version of the submodule, and if
the submodule we get is out of date then the submodule update will
fail anyway.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoadhoc-revtuple-generator: Do not always reclone
Ian Jackson [Fri, 27 Jun 2014 14:24:18 +0000 (15:24 +0100)]
adhoc-revtuple-generator: Do not always reclone

Instead, try setting the `origin' remote's URL and fetching.  If that
fails it may be that the local repo is corrupted, so then we try
recloning.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoadhoc-revtuple-generator: Avoid an undefined variable warning
Ian Jackson [Fri, 27 Jun 2014 13:44:58 +0000 (14:44 +0100)]
adhoc-revtuple-generator: Avoid an undefined variable warning

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoadhoc-revtuple-generator: Include $treename in some error messages
Ian Jackson [Fri, 27 Jun 2014 11:30:35 +0000 (12:30 +0100)]
adhoc-revtuple-generator: Include $treename in some error messages

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoadhoc-revtuple-generator: Cope with parentless git commits
Ian Jackson [Fri, 27 Jun 2014 11:30:13 +0000 (12:30 +0100)]
adhoc-revtuple-generator: Cope with parentless git commits

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agots-xen-install: Correct mlog to logm
Ian Campbell [Mon, 16 Jun 2014 08:31:03 +0000 (09:31 +0100)]
ts-xen-install: Correct mlog to logm

Only logm exists.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agomg-debian-installer-update: Print messages to stderr.
Anthony PERARD [Mon, 9 Jun 2014 15:16:50 +0000 (16:16 +0100)]
mg-debian-installer-update: Print messages to stderr.

This fix standalone-reset which rely on the output of this script.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Dario Faggioli <dario.faggioli@citrix.com>
10 years agomg-hosts: Usage documentation.
Ian Jackson [Wed, 25 Jun 2014 14:05:34 +0000 (15:05 +0100)]
mg-hosts: Usage documentation.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomg-hosts: Allow spaces in setflagexpr exprs
Ian Jackson [Wed, 25 Jun 2014 14:05:21 +0000 (15:05 +0100)]
mg-hosts: Allow spaces in setflagexpr exprs

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-for-branches: clean up tmp/
Ian Jackson [Wed, 25 Jun 2014 09:33:01 +0000 (10:33 +0100)]
cr-for-branches: clean up tmp/

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoImplement direct IPMI power control.
Ian Campbell [Mon, 16 Jun 2014 08:51:21 +0000 (09:51 +0100)]
Implement direct IPMI power control.

The xenuse module currently used to control some machines (specifically the ARM
marilith machines) is unreliable and subject to breakages on the xenrt server
side.

Since xenuse/xenrt is just wrapping IPMI for these machines implement direct
control via a new Osstest::PDI::impi module.

This is distinct from the existing impiextra (arbitrary ipmi commands on power
status change) because IPMI power control is not idempotent, therefore it is
necessary to query the current status before turning the machine on or off.

After this change the correct power method for a marilith node is:
    ipmiextra on marilith-n0-mgmt.uk.xensource.com <user> <pass> chassis bootdev pxe;ipmi marilith-n0-mgmt.uk.xensource.com <user> <pass>

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
Ian, when we discussed this on IRC you expressed a preference for using
freeipmitools, but since the existing ipmiextra module uses ipmitool I figured
it would be better to be consistent.

10 years agocrontab: reduce frequency of linux-next
Ian Jackson [Mon, 9 Jun 2014 14:29:52 +0000 (15:29 +0100)]
crontab: reduce frequency of linux-next

10 years agomake-flight: Do not filter the qemu-upstream-* jobs
Ian Jackson [Mon, 9 Jun 2014 14:20:05 +0000 (15:20 +0100)]
make-flight: Do not filter the qemu-upstream-* jobs

When "qemu-xen-traditional" was the default, this filtering was
correct.  But now, "qemu-xen" is the default, and that version may
also be used in other tests for ancillary functions (eg, in dom0 for
the bootloader).

So simply do all of the jobs when testing qemu-upstream-*.

Retain the old behaviour for qemu-xen-4.2-testing, which has
qemu-xen-traditional as the default.  (There is no
qemu-xen-4.1-testing, and 4.3 has the change to the default.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agobuild_clone: When guessing vcs from url suffix, strip cache info
Ian Jackson [Thu, 5 Jun 2014 10:15:46 +0000 (11:15 +0100)]
build_clone: When guessing vcs from url suffix, strip cache info

We assume that anything not containing `/'s after `%20' is
uninteresting.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoap-common: Use besteffort_repo for rumpsrc
Ian Jackson [Wed, 4 Jun 2014 13:47:54 +0000 (14:47 +0100)]
ap-common: Use besteffort_repo for rumpsrc

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocri-common, ap-common: abstract out besteffort_repo
Ian Jackson [Wed, 4 Jun 2014 13:46:49 +0000 (14:46 +0100)]
cri-common, ap-common: abstract out besteffort_repo

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-daily-branch: Fix URLs used for trees which should be using push gate output
Ian Campbell [Wed, 4 Jun 2014 12:49:16 +0000 (13:49 +0100)]
cr-daily-branch: Fix URLs used for trees which should be using push gate output

For several trees we were using the upstream URL for tests of all branches when
we should have been using the push gate output on xenbits (to help isolate us
from external failures). i.e. tree_libvirt was git://libvirt.org/libvirt.git
for every branch but for branches other than libvirt it should be
git://xenbits.xen.org/libvirt.git.

This is down to a missing third argument to determine_version. Fix this for all
but osstest (which shouldn't be touched).

Tested with:
 $ BRANCHES="xen-unstable libvirt rumpuserxen seabios qemu-mainline qemu-xen-unstable linux-next linuxfirmware osstest"
 $ for b in $BRANCHES ; do f=T${b//-}; ./standalone make-flight -f $f $b; done
 $ check() { \
     f=T${1//-} ; \
     echo $1 ; \
     ./mg-show-flight-runvars $f | \
         awk '$2 ~ /^tree_/ && $3 !~ /^$|xenbits.xen.org/ { printf "    %-28s %s\n", $2, $3 }' | \
         sort -u; \
   }
 $ for b in xen-unstable libvirt rumpuserxen seabios qemu-mainline qemu-xen-unstable linux-next osstest ; do check $b; done

Which produces:

xen-unstable
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
libvirt
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_libvirt                 git://libvirt.org/libvirt.git
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
rumpuserxen
    tree_rumpuserxen             https://github.com/rumpkernel/rumpuser-xen
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
seabios
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
    tree_seabios                 git://git.seabios.org/seabios.git
qemu-mainline
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_qemuu                   git://git.qemu.org/qemu.git
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
qemu-xen-unstable
linux-next
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_linux                   git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
linuxfirmware
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src
osstest
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src

The presence of tree_gnulib_libvirt is expected while the
tree_rumpuserxen_rumpsrc is erroneous (I think) but not intended to be fixed
here.

Note that linuxfirmware is broken since tree_linuxfirmware should be
$GIT_KERNEL_ORG/pub/scm/linux/kernel/git/firmware/linux-firmware.git. This is
also not addressed here.

For comparison the xen-unstable check output prior to this fix was:

xen-unstable
    tree_gnulib_libvirt          git://drall.uk.xensource.com:9419/git://git.sv.gnu.org/gnulib.git%20[fetch=try]
    tree_libvirt                 git://libvirt.org/libvirt.git
    tree_rumpuserxen             https://github.com/rumpkernel/rumpuser-xen
    tree_rumpuserxen_rumpsrc     https://github.com/rumpkernel/rumpkernel-netbsd-src

Note the incorrect tree_libvirt and tree_rumpuserxen.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson <ian.jackson@eu.citrix.com

10 years agomake-flight: include cri-common before ap-common
Ian Campbell [Wed, 4 Jun 2014 11:05:56 +0000 (12:05 +0100)]
make-flight: include cri-common before ap-common

The latter wants cached_repo which the former provides.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson <ian.jackson@eu.citrix.com

10 years agostandalone: ensure log dir exists for all users of with_logging
Ian Campbell [Wed, 4 Jun 2014 11:00:02 +0000 (12:00 +0100)]
standalone: ensure log dir exists for all users of with_logging

By moving the check into that function. Previously at least make-flight was
failing to all ensure_logs

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson <ian.jackson@eu.citrix.com

10 years agots-logs-capture: collect /proc/partitions
Ian Jackson [Wed, 4 Jun 2014 11:29:36 +0000 (12:29 +0100)]
ts-logs-capture: collect /proc/partitions

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agots-logs-capture: collect resize2fs.log
Ian Jackson [Wed, 4 Jun 2014 11:28:48 +0000 (12:28 +0100)]
ts-logs-capture: collect resize2fs.log

This is generated by ts-xen-build-prep but previously we didn't
collect it.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agots-xen-build-prep: Run the backgrounded resize2fs with ionice
Ian Jackson [Wed, 4 Jun 2014 11:24:06 +0000 (12:24 +0100)]
ts-xen-build-prep: Run the backgrounded resize2fs with ionice

We wrap this in
  ionice -cBest-Effort -n7

This is an attempt to stop resize2fs from starving builds of IO
bandwidth.  We have been seeing that some IO-intensive build steps
(notably, configure) can take unreasonable amounts of time and we
suspect resize2fs as the culprit.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoREADME.dev: more info about mg-blockage
Ian Jackson [Tue, 3 Jun 2014 15:08:49 +0000 (16:08 +0100)]
README.dev: more info about mg-blockage

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoTolerate outages of git://git.sv.gnu.org/gnulib.git
Ian Campbell [Tue, 20 May 2014 10:54:36 +0000 (11:54 +0100)]
Tolerate outages of git://git.sv.gnu.org/gnulib.git

gnulib is a bit special in that it isn't part of a push gate and
therefore doesn't have an output repo on xenbits. This means that if
git.sv.gnu.org is down for some reason then any flight which uses it
will potentially fail, even ones which are not specifically trying to
test gnulib.git. Normally unrelated flights are isolated from 3rd
party failures because they use the relevant push gates' outputs.

The git-proxy which we use supports passing [fetch=try] as an option
which will cause it to try and fetch from upstream and fall back to
the cached version if upstream does not respond. This is exactly what
we want for gnulib.git. Proxy options in the URL are separate with a
space which we encode here as %20 to avoid having to go around adding
quotes to all the uses of shell variables which contain a URL.

Provide a helper shell function cached_repo which conditionally
arranges for the given repo to be cached with the given options only
if the osstest config sets GitCacheProxy, so as not to break uses of
osstest outside of the production environment where a cache may not be
available (of course such uses will then be subject to outages of 3rd
party repos)

It was also necessary to make repo_tree_rev_fetch_git idempotent WRT
the proxy prefix to avoid trying to clone
git://proxy/git://proxy/git://some.real.domain/foo.git.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
10 years agocri-args-hostlists, etc.: Send rumpuserxen reports to rumpkernel-builds
Ian Jackson [Tue, 3 Jun 2014 14:59:51 +0000 (15:59 +0100)]
cri-args-hostlists, etc.: Send rumpuserxen reports to rumpkernel-builds

There is now a list, rumpkernel-builds@lists.sourceforge, for this
kind of information relating to rumpuser-xen.

Provide a facility in cri-args-hostlists for start_email to look for
$OSSTEST_EMAIL_HEADER--$branch.

Introduce two new files
  daily-cron-email-real--rumpuserxen
  daily-cron-email-real-bisectcomplete--rumpuserxen
which override the general settings.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocs-bisection-step: Remove a bit of duplication
Ian Jackson [Tue, 3 Jun 2014 14:52:18 +0000 (15:52 +0100)]
cs-bisection-step: Remove a bit of duplication

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocs-bisection-step: Cope with url-less trees
Ian Jackson [Mon, 2 Jun 2014 10:09:17 +0000 (11:09 +0100)]
cs-bisection-step: Cope with url-less trees

Since osstest 64a9c064, some flights have a tree with an empty
tree_<foo> variable.  This breaks the bisector.  Make it more robust
as follows:

When searching for the basis flight, ignore not only trees with no url
specified but also ones with an empty url.

When preparing the set of trees we care about, eliminate any where the
latest flight has no url (or an empty url).

(It would still be better for the flight constructor to leave
tree_<foo> unset, and for the ts-* build script to set it in as a part
of the post-build data collection.  That would allow the bisector to
work on that tree where relevant.  But doing this is a task for
another day - if for no other reason than that we have to decide
whether to put it in a new variable built_tree_<foo>.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoap-qemu-revision: Make it actually work
Ian Jackson [Fri, 23 May 2014 09:55:51 +0000 (10:55 +0100)]
ap-qemu-revision: Make it actually work

This is used (only) for obtaining lists of the corresponding qemu
revisions (for the report email) when doing a Xen push.  It has never
worked because we didn't provide a type argument to git-cat-file.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-daily-branch: Only set TREE_SEABIOS for seabios flights
Ian Campbell [Fri, 23 May 2014 09:26:26 +0000 (10:26 +0100)]
cr-daily-branch: Only set TREE_SEABIOS for seabios flights

For all other flights we want to use the Config.mk settings.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoDefault vcs to git
Ian Jackson [Thu, 22 May 2014 11:24:19 +0000 (12:24 +0100)]
Default vcs to git

adhoc-revtuple-generator and cs-bisection-step try to guess the VCS
from the URL, by looking at the end of the URL for a suitable file
extension.  If none was found, things would break.

This relies on a convention about tree URLs which is sadly no longer
being widely observed.  In particular many projects have git URLs
which do not mention git.

Instead, have these scripts assume that urls which don't end in .git
or .hg are for git.

This renders hg into a second-class VCS in osstest, which is (I think)
tolerable.  If it isn't, then someone ought to come up with some
better way of ensuring that the vcs is explicitly specified to
adhoc-revtuple-generator and cs-bisection-step.  That would probably
be doable but is too much work right now.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agosg-run-job: Do blocked check globally, in run-job
Ian Jackson [Wed, 21 May 2014 11:36:02 +0000 (12:36 +0100)]
sg-run-job: Do blocked check globally, in run-job

Only run-job can fully abandon the whole job.  With the previous
arrangements (in bc63a901), a blocked build job would go on to try to
run ts-logs-capture and become broken (rather than blocked).

The downside of this is that it is not possible in the future to
introduce a kind of job which can selectively carry on if some of its
dependencies fail.  This notion is IMO an acceptable casualty.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoTestSupport::broken: Update status of `preparing' jobs
Ian Jackson [Wed, 21 May 2014 11:34:01 +0000 (12:34 +0100)]
TestSupport::broken: Update status of `preparing' jobs

If the current job is preparing, and we call broken, then we should
update the status to that supplied by broken's caller.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomake-flight: rumpuserxen amd64 too
Ian Jackson [Wed, 21 May 2014 10:54:10 +0000 (11:54 +0100)]
make-flight: rumpuserxen amd64 too

Build and test amd64 rump kernel.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocrontab: Rearrange actually-daily branches
Ian Jackson [Wed, 21 May 2014 10:37:48 +0000 (11:37 +0100)]
crontab: Rearrange actually-daily branches

Make them start at roughly the same time, by using cr-for-branches for
them all in one go.  That increases the extent to which build hosts
will be reused (saving on reinstallations), and hence also the extent
to which the build hosts' ccache will be effective (saving on cpu).

Set them off at 04:18 UK time (currently equivalent to 21:18 Pacific
time), in the hope that at least some of them might have produced
answers during following the UK working day.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agorumpuserxen: Increase build timeout for grievous bodge
Ian Jackson [Wed, 21 May 2014 10:42:37 +0000 (11:42 +0100)]
rumpuserxen: Increase build timeout for grievous bodge

This involves copying 90Mby of data.  If the test machine is busy (in
the worst case, running the kernel side of resize2fs which due to
Linux's poor IO scheduling hogs the entire IO system) it can take a
long time.  In flight 26359 it was still running after 280s, when the
machine was released.

Increase the timeout to 2000s.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agocr-publish-flight-logs: Provide ref path for osstest branch push
Ian Jackson [Mon, 19 May 2014 10:58:47 +0000 (11:58 +0100)]
cr-publish-flight-logs: Provide ref path for osstest branch push

Without this, the push fails if the osstest tree is in detached head
state.  (Which is a situation which can occur during by-hand testing.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoAdd my developer/deployment notes as README.dev
Ian Campbell [Mon, 19 May 2014 11:08:09 +0000 (12:08 +0100)]
Add my developer/deployment notes as README.dev

These are mostly targetted at people setting up things on the production
infra.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoproduction-config: Update to pickup latest Debian Installer
Ian Campbell [Mon, 19 May 2014 11:32:16 +0000 (12:32 +0100)]
production-config: Update to pickup latest Debian Installer

Pickup the changes made in
     mg-debian-installer-update: Updates for newer armhf kernels from backports.
     mg-debian-installer-update: grab Xen PV domU capable images too

I have run

    for i in i386 amd64 armhf ; do
        OSSTEST_CONFIG=production-config ./mg-debian-installer-update wheezy $i 'firmware-bnx2'
    done

on the osstest vm. Point the config to these new versions.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agosg-run-job: check for prerequisite builds when running build jobs
Ian Campbell [Thu, 15 May 2014 10:30:39 +0000 (11:30 +0100)]
sg-run-job: check for prerequisite builds when running build jobs

For example build-$ARCH-libvirt depends on build-$ARCH for the Xen bits.
Performing this check early saves allocating, installing, preparating etc a
host just to find the prerequisite is missing.

Rename ts-xen-build-check to ts-build-check while we are there. The script
checks all prerequisite builds, not just Xen.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agostandalone: allow making a baseline flight.
Ian Campbell [Thu, 15 May 2014 10:30:38 +0000 (11:30 +0100)]
standalone: allow making a baseline flight.

While there make the existing --lvextend argument actually work (by passing it
to getopt) and document both.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agots-xen-build: Move Xen build tree from xen-unstable to just xen
Ian Campbell [Thu, 15 May 2014 10:30:37 +0000 (11:30 +0100)]
ts-xen-build: Move Xen build tree from xen-unstable to just xen

It's not always xen-unstable we are testing, and I had to check that it was
the right generic name, so reduce this confusion.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agots-xen-build: stash the Xen and SeaBIOS .config's
Ian Campbell [Thu, 15 May 2014 10:30:36 +0000 (11:30 +0100)]
ts-xen-build: stash the Xen and SeaBIOS .config's

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoAdd a flight to test seabios.org's master branch
Ian Campbell [Thu, 15 May 2014 10:30:35 +0000 (11:30 +0100)]
Add a flight to test seabios.org's master branch

Most of this is the usual stuff for adding a new branch.

cr-daily-branch needed to set REVISION_SEABIOS for the seabios branch and leave
it blank for others (so that Config.mk's version will be used). ts-xen-build
already consumes the resulting version runvar but needed to be taught about the
tree URL runvar.

make-flight and mfi-common are updated to only build+test seabios on x86.

The new branch is added to cr-daily-branches.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoAdd a flight to test qemu.org's ("mainline") master branch.
Ian Campbell [Thu, 15 May 2014 10:30:34 +0000 (11:30 +0100)]
Add a flight to test qemu.org's ("mainline") master branch.

The naming here is slightly confused because we call our branches of qemuu on
xenbits "qemu-upstream-foo", so I went with qemu-mainline to be distinct.

I've introduced $qemuubranch along the lines of $xenbranch, it is normally set
to qemu-upstream-{X.Y,unstable} but for the qemu-mainline flight it instead
points to qemu-mainline.

I've run up to the make-flight bit of cr-daily-branch and it differs only in
the expected ways from a standard xen-unstable branch (i.e. uses xen.git#master
and qemu.git#master).

I did consider causing make-flight:job_create_test_filter_callback to omit any
test which didn't use qemuu but I decided not to because it is used for PV
qdisk backends too.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agomg-debian-installer-update: grab Xen PV domU capable images too
Ian Campbell [Mon, 28 Apr 2014 10:41:24 +0000 (11:41 +0100)]
mg-debian-installer-update: grab Xen PV domU capable images too

Some test cases are going to want this, so they should be grabbed and
versioned here.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agoBuild against liblzma to support .xz compressed kernels
Ian Campbell [Mon, 28 Apr 2014 10:41:23 +0000 (11:41 +0100)]
Build against liblzma to support .xz compressed kernels

Install the library at runtime.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agocr-external-linux: remove
Ian Campbell [Mon, 28 Apr 2014 10:41:22 +0000 (11:41 +0100)]
cr-external-linux: remove

Seems to be unused.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agocr-daily-branch: Only build oldkern for xen-unstable flights
Ian Campbell [Thu, 15 May 2014 10:35:44 +0000 (11:35 +0100)]
cr-daily-branch: Only build oldkern for xen-unstable flights

This is just supposed to be a build test, it doesn't actually get used.
There's no need to be doing this for every flight, doing it for xen-unstable
is enough

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agocrontab: Bisect the libvirt branch
Ian Campbell [Wed, 14 May 2014 08:14:16 +0000 (09:14 +0100)]
crontab: Bisect the libvirt branch

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
10 years agomg-debian-installer-update: Updates for newer armhf kernels from backports.
Ian Campbell [Mon, 28 Apr 2014 09:18:38 +0000 (10:18 +0100)]
mg-debian-installer-update: Updates for newer armhf kernels from backports.

In newer kernels sd_mod.ko requires crypto symbols, so include
kernel/crypto/*.ko in the installer overlay cpio.

Also prefer the newest versions of packages pulled in from backports, by
reversing the sort. Anchor the linux-image package name to we only pick up the
-armmp flavour and not -armmp-lpae (which is unnecessary in an installer
environment).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agorump kernels: Create a "branch" for rump kernel tests
Ian Jackson [Fri, 16 May 2014 16:22:10 +0000 (17:22 +0100)]
rump kernels: Create a "branch" for rump kernel tests

This is a traditional push gate.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agorump kernels: Provide two test jobs
Ian Jackson [Fri, 16 May 2014 15:57:16 +0000 (16:57 +0100)]
rump kernels: Provide two test jobs

10 years agorump kernels: Provide a build job
Ian Jackson [Thu, 15 May 2014 15:48:56 +0000 (16:48 +0100)]
rump kernels: Provide a build job

Provide:
  - test script ts-rumpuserxen-build
  - recipe build-rumpuserxen
  - in flights, job build-i386-rumpuserxen
  - in ap-common, TREE and TREEVCS for RUMPUSERXEN and RUMPUSERXEN_RUMPSRC

mfi-common also honours REVISION_RUMPUSERXEN, but provides no way to
specify the revisions of the subtrees (the two submodules and
rumpsrc).  The upstream rumpuser-xen git repository contains the
commit hashes of those.

We had to do some pretty unpleasant things to get the rumpuser-xen
build system to work with rumpsrc we cloned ourselves.

During bisection the bisector must choose a unification of the various
trees' revision graphs, into a single graph.  But we do not here teach
adhoc-revtuple-generator about these interrelationships.  As a result
the bisector may choose a unification inconsistent with the subtree
references in rumpuser-xen.git.  This might impair the performance of
the bisector in the presence of cross-tree version incompatibilities,
but the results should be tolerable for now.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agomfi-common: Honour REVISION_LIBVIRT=disable
Ian Jackson [Fri, 16 May 2014 16:56:54 +0000 (17:56 +0100)]
mfi-common: Honour REVISION_LIBVIRT=disable

This makes it possible for (for example) cr-daily-branch to disable
the libvirt build, when it is not needed.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
10 years agoprepareguest: Tolerate $mb=undef
Ian Jackson [Fri, 16 May 2014 15:51:59 +0000 (16:51 +0100)]
prepareguest: Tolerate $mb=undef

Allow callers to pass undef for $mb, to indicate that there is no need
for prepareguest to do anything about storage.

No functional change for any of the existing callers.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>