]> xenbits.xensource.com Git - people/liuw/osstest.git/log
people/liuw/osstest.git
9 years agotoolstack/libvirt: install libnl-3-200 on Jessie wip.debian-jessie-4
Wei Liu [Thu, 18 Jun 2015 14:46:07 +0000 (15:46 +0100)]
toolstack/libvirt: install libnl-3-200 on Jessie

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
9 years agots-kernel-build: enable CONFIG_FHANDLE
Wei Liu [Fri, 12 Jun 2015 22:50:55 +0000 (23:50 +0100)]
ts-kernel-build: enable CONFIG_FHANDLE

According to Debian bug #764539, systemd needs this kernel feature to be
able to mount disks. Complete list of required kernel features can be
found in /usr/share/doc/systemd/README.gz.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agots-xen-install: install libnl-route-3-200 for jessie
Wei Liu [Fri, 12 Jun 2015 16:53:50 +0000 (17:53 +0100)]
ts-xen-install: install libnl-route-3-200 for jessie

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agots-xen-build-prep: reverse the test for installing libc6-dev-i386
Wei Liu [Wed, 20 May 2015 17:56:28 +0000 (18:56 +0100)]
ts-xen-build-prep: reverse the test for installing libc6-dev-i386

Starting from wheezy, Debian introduced multiarch support, so we need to
install libc6-dev-i386 to build tools.

Since multiarch will be permanent, we reverse the test to not install
libc6-dev-i386 on releases older than wheezy (i.e. wheezy and jessie
will have that package).

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agots-xen-build-prep: install libnl-route-3-dev on Jessie
Wei Liu [Wed, 20 May 2015 17:56:27 +0000 (18:56 +0100)]
ts-xen-build-prep: install libnl-route-3-dev on Jessie

There is no package called libnl-dev on Jessie.  To build remus, use
libnl-route-3-dev instead.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoDebian: grub2: Allow submenu and menuentry items to be indented.
Ian Campbell [Thu, 28 May 2015 08:24:51 +0000 (09:24 +0100)]
Debian: grub2: Allow submenu and menuentry items to be indented.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoCope with Jessie's d-i vg name
Wei Liu [Mon, 15 Jun 2015 11:07:38 +0000 (12:07 +0100)]
Cope with Jessie's d-i vg name

In Jessie the default vg name is changed to "$hostname-vg". Make that
default case and check for wheezy, squeeze and lenny for backward
compatibility.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v4:
1. Add missing "use Osstest::Debian;".

Changes in v3:
1. Introduce a function to return vg name and use it.

Changes in v2:
1. Make Jessie case default.

9 years agoDebian.pm: install grub to default device
Wei Liu [Wed, 20 May 2015 17:56:25 +0000 (18:56 +0100)]
Debian.pm: install grub to default device

Debian installer used to have the behaviour to install grub on the
default device. But it's changed in Jessie.

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759737.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoForce sysvinit with Jessie
Ian Campbell [Fri, 12 Jun 2015 10:24:11 +0000 (11:24 +0100)]
Force sysvinit with Jessie

Systemd  doesn't honor osstest-confirm-booted service, which breaks
ts-leak-check.  Fall back to SysV init for now.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
[wei: rewrite commit message and code comment]
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomg-debian-installer-update: Also collect DTBs for standard kernels
Wei Liu [Tue, 16 Jun 2015 16:10:00 +0000 (17:10 +0100)]
mg-debian-installer-update: Also collect DTBs for standard kernels

For armhf and arm64 for Jessie we will need these in the normal case
as well as in the backports case. Arrange to download.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
[wei: fix rebase conflict]
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomg-debian-installer-update: updates to better handle Jessie onwards.
Ian Campbell [Mon, 26 Jan 2015 16:31:35 +0000 (16:31 +0000)]
mg-debian-installer-update: updates to better handle Jessie onwards.

In mg-debian-installer-update:

  - Expand the list of (suite,arch) combinations which don't exist and
    move it to the top.

  - Tweak the backports.org handling to allow it to be specified on a
    per (suite,arch) basis, and specify it only for (wheezy,armhf)
    since it is not currently needed for jessie. Make the
    initramfs-tools fetching conditional too.

  - Only extract backports-dtbs if they are present in the package.

  - Handle arm64.

  - Collect the correct armhf kernel for Jessie onwards (vexpress
    removed from path).

Have mg-debian-installer-update-all fetch arm64 too.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
[wei: wrap lines]
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoMore flexible handling of need-kernel-deb-$flavour host flag
Ian Campbell [Mon, 26 Jan 2015 16:08:43 +0000 (16:08 +0000)]
More flexible handling of need-kernel-deb-$flavour host flag

The current behaviour does not account for the need to apply this
workaround only for particular Debian releases, i.e. Midway will be
supported just fine in Jessie.

It also does not allow for kernels from alternative sources other than
backports (e.g. experimental, although that is hypothetical right
now).

The flavour name is actually irrelevant in practice (none of our
supported arch's need multiple kernel flavours at install time, which
is all we care about). So omit it from the new scheme.

The new scheme is "need-kernel-deb-$suite-$altsource", i.e.
need-kernel-deb-wheezy-backports. Possibly in the future e.g.
need-kernel-deb-jessie-experimental indicating that the experimental
kernel is needed on jessie.

mg-debian-installer-update is modified to download to the new name.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agocri-args-hostlists: Run sg-report-host-history
Ian Jackson [Wed, 17 Jun 2015 16:41:49 +0000 (17:41 +0100)]
cri-args-hostlists: Run sg-report-host-history

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agosg-report-host-history: Make --html-dir have to be host/
Ian Jackson [Wed, 17 Jun 2015 16:38:37 +0000 (17:38 +0100)]
sg-report-host-history: Make --html-dir have to be host/

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agosg-report-host-history: Show "running" jobs as "incomplete"
Ian Jackson [Wed, 17 Jun 2015 16:35:50 +0000 (17:35 +0100)]
sg-report-host-history: Show "running" jobs as "incomplete"

Since they may in fact be abandoned.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agosg-report-host-history: Move per-row endedq query into per-host transaction
Ian Jackson [Wed, 17 Jun 2015 16:35:19 +0000 (17:35 +0100)]
sg-report-host-history: Move per-row endedq query into per-host transaction

No substantial change, but reduces the work done in the main
transaction.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agosg-report-host-history: Aggregate runvars query for all hosts
Ian Jackson [Wed, 17 Jun 2015 16:21:10 +0000 (17:21 +0100)]
sg-report-host-history: Aggregate runvars query for all hosts

This is much faster.  It might short-change unpopular hosts rather;
hence the change of the limit fudge factor from 2 to 3.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agosg-report-host-history: Support flight:FLIGHT
Ian Jackson [Wed, 17 Jun 2015 16:05:58 +0000 (17:05 +0100)]
sg-report-host-history: Support flight:FLIGHT

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agosg-report-host-history: Use a hash for hosts
Ian Jackson [Wed, 17 Jun 2015 15:57:39 +0000 (16:57 +0100)]
sg-report-host-history: Use a hash for hosts

This will allow deduplication.  No functional change other than a
change to the order of processing.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agosg-report-host-history: Move database manipulations
Ian Jackson [Wed, 17 Jun 2015 15:55:26 +0000 (16:55 +0100)]
sg-report-host-history: Move database manipulations

Arrange for the SET LOCAL to have the right scope.  Run
computeflightsrange, and reporthost, each within a db transaction,
with an appropriate lock.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agosg-report-host-history: Move query preparation into jobquery
Ian Jackson [Tue, 16 Jun 2015 13:54:44 +0000 (14:54 +0100)]
sg-report-host-history: Move query preparation into jobquery

To get the scope of database locking, and of the SET LOCAL workaround,
right, it is convenient to move these query preparations into the
jobquery function.

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agosg-report-host-history: Break out computeflightsrange
Ian Jackson [Tue, 16 Jun 2015 10:49:32 +0000 (11:49 +0100)]
sg-report-host-history: Break out computeflightsrange

To do database locking coherently, it will be convenient to have this
in a function.

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agosg-report-host-history: Introduce new script
Ian Jackson [Fri, 29 May 2015 15:26:16 +0000 (15:26 +0000)]
sg-report-host-history: Introduce new script

This script is still a work in progress.  It does currently mostly
work and can generate a useful report.  However, it needs:

 * Some database locking (to avoid deadlock errors)
 * A --flight= option to make it automatically report host usage
   for all hosts touched in a flight, for the use of cr-*
 * Hooking into cr-*

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Improved commit message.
    Fix a debug bug.
    Actually install the html; do not leave it as .new

9 years agoreporting: sg-report-flight should ignore missing jobs
Ian Jackson [Wed, 17 Jun 2015 13:42:56 +0000 (13:42 +0000)]
reporting: sg-report-flight should ignore missing jobs

The function findaflight should not, when the caller specifies a job,
find a flight which does not contain that job at all.

One effect of allowing it to find such flights is that it might find a
bisection flight and try to use it as a basis for claiming a
regression, or as a justification for something not being a
regression, and then complain that all the missing steps in the
bisection flight are `blocked'.

This can be seen in the report for 58627:
 test-amd64-i386-xl-qemuu-winxpsp3  6 xen-boot fail blocked in 56366-bisect

After this patch, a report generated for 58627 no longer mentions
56366 at all.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoreporting: Move bisection report outputs
Ian Jackson [Tue, 16 Jun 2015 15:38:23 +0000 (16:38 +0100)]
reporting: Move bisection report outputs

Replace some `.'s in filenames with `/'s, making the directory
listings easier to deal with.

(I haven't been able to conveniently do a proper test of this change,
but I have dry-run the critical parts.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoreporting: Move job histories into history/
Ian Jackson [Tue, 16 Jun 2015 14:06:36 +0000 (15:06 +0100)]
reporting: Move job histories into history/

Replace some `.'s in HTML filenames with `/'s, making the directory
listings easier to deal with.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoreporting: Show slightly better info for broken jobs
Ian Jackson [Fri, 29 May 2015 17:46:53 +0000 (17:46 +0000)]
reporting: Show slightly better info for broken jobs

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoreporting: Add colours as optional export tag, and provide $blue
Ian Jackson [Fri, 29 May 2015 15:20:21 +0000 (15:20 +0000)]
reporting: Add colours as optional export tag, and provide $blue

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoreporting: Move report_run_getinfo and some colours into Executive.pm
Ian Jackson [Fri, 29 May 2015 15:20:03 +0000 (15:20 +0000)]
reporting: Move report_run_getinfo and some colours into Executive.pm

This is to support a forthcoming sg-report-host-history.

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agosg-report-job-history: Show start time
Ian Jackson [Fri, 29 May 2015 14:24:53 +0000 (14:24 +0000)]
sg-report-job-history: Show start time

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agosg-report-job-history: Cope if history too short
Ian Jackson [Fri, 29 May 2015 14:14:41 +0000 (14:14 +0000)]
sg-report-job-history: Cope if history too short

If there have been less than 99 relevant flights, the inner SELECT (to
determine the minimum flight number) would return NULL.  And anything
> NULL is NULL and NULL is treated as false.  So the host runvar
identification would break.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agosg-report-job-history: Slightly prettify sql
Ian Jackson [Fri, 29 May 2015 14:13:34 +0000 (14:13 +0000)]
sg-report-job-history: Slightly prettify sql

No functional change apart from slightly better debug output.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agosg-report-job-history: Add a debugging statement
Ian Jackson [Fri, 29 May 2015 13:26:10 +0000 (13:26 +0000)]
sg-report-job-history: Add a debugging statement

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agosg-report-flight: Use db_prepare
Ian Jackson [Fri, 29 May 2015 13:24:13 +0000 (13:24 +0000)]
sg-report-flight: Use db_prepare

And do away with two commented-out ad-hoc statement dumps.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocs-bisection-step: Use db_prepare
Ian Jackson [Fri, 29 May 2015 13:22:44 +0000 (13:22 +0000)]
cs-bisection-step: Use db_prepare

And do away with one ad-hoc statement dump.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agosg-report-job-history: Use db_prepare
Ian Jackson [Fri, 29 May 2015 13:20:13 +0000 (13:20 +0000)]
sg-report-job-history: Use db_prepare

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoOsstest.pm: Provide new db_prepare helper with built-in debugging
Ian Jackson [Fri, 29 May 2015 13:19:39 +0000 (13:19 +0000)]
Osstest.pm: Provide new db_prepare helper with built-in debugging

No callers, so no functional change, as yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Suppress "used only once" warning

9 years agoArrange to upgrade microcode on x86 test hosts.
Ian Campbell [Mon, 15 Jun 2015 08:36:02 +0000 (09:36 +0100)]
Arrange to upgrade microcode on x86 test hosts.

Both Xen and Linux support extracting a microcode update from an
initramfs early during boot. This requires prepending a suitable
uncompressed cpio archive containing the necessary files to the
initrd.

Xen also supports loading the microcode cpio from any multiboot
module, but for in order to allow the possibility of loading on native
boots (e.g. for build jobs) we prefer the prepend method.

This patch provides mg-cpu-microcode-update which creates a suitable
microcode cpio in the images directory and arranges for it to be added
to the host during installation (so it is done before the kernel is
installed and initramfs generated etc, saving faff). It also adds
"ucode=scan" to the Xen command line when necessary.

The version of initramfs-tools in Wheezy does not yet support
prepending things to the initrm, so we use a custom compression
command which sneakily does it for us. This could be done better from
Jessie onwards.

Note that Linux only supports this from v3.8 onwards, so this doesn't
work for Wheezy (which uses v3.2). From Jessie onwards we should
benefit from microcode updates even for native (build) jobs.

Tested on both Intel and AMD where it appeared to have the desired
effect under Xen. Under native I only tried Wheezy which doesn't
support early microcode.

Note that I've not bothered to implement late microcode updates, which
would work on v3.2 too, since that would need a different set of files
etc and it doesn't seem especially pressing.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Jan Beulich <JBeulich@suse.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoStubdom test case
Wei Liu [Mon, 15 Jun 2015 13:52:37 +0000 (14:52 +0100)]
Stubdom test case

Currently only QEMU traditional supports stubdom and we want to
eventually do away with non-XSM test cases, so we only create

  test-amd64-{i386,amd64}-xl-qemut-stubdom-debianhvm-amd64-xsm

Note that stubdom only supports serial='pty'. Piping serial to stderr
causes stubdom to exit abnormally.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoTestSupport: introduce guest_var_boolean
Wei Liu [Mon, 15 Jun 2015 13:52:36 +0000 (14:52 +0100)]
TestSupport: introduce guest_var_boolean

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoosstest: reduce FreeBSD install timeouts
Roger Pau Monne [Fri, 29 May 2015 10:38:18 +0000 (11:38 +0100)]
osstest: reduce FreeBSD install timeouts

Only the first block is expected to take longer (because it decompresses
the image and writes it to a LVM volume), the remaining commands should
execute much faster, so reduce the timeout.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agots-libvirt-build: initscript should depend on xenstored service
Ian Campbell [Tue, 26 May 2015 11:36:00 +0000 (12:36 +0100)]
ts-libvirt-build: initscript should depend on xenstored service

In order that xencommons is guarenteed to have been started before
libvirtd. Otherwise sometimes libvirt can be started first resulting
in:
error: invalid argument: unsupported config type xen-xl

Because xen wasn't available when libvirt started.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoIgnore guest-stop failures for windows 7 tests.
Ian Campbell [Tue, 9 Jun 2015 09:54:00 +0000 (10:54 +0100)]
Ignore guest-stop failures for windows 7 tests.

As described in <1432284841.10746.136.camel@citrix.com> /
http://lists.xen.org/archives/html/xen-devel/2015-05/msg03016.html
Windows 7 does not appear to reliably actually shutdown when asked to
via the ACPI power button.

This test step has been failing forever until the "Toolstack::xl:
Support for ACPI fallback for shutdown" patch to osstest, and now it
passes only very infrequently, so we may as well ignore it.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Andrew Cooper <Andrew.Cooper3@citrix.com>
Cc: Paul Durrant <paul.durrant@citrix.com>
Cc: Ian Jackson <Ian.Jackson@eu.citrix.com>,
Cc: Jan Beulich <JBeulich@suse.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoToolstack: Do not pass -F to xm shutdown
Ian Campbell [Mon, 1 Jun 2015 08:32:37 +0000 (09:32 +0100)]
Toolstack: Do not pass -F to xm shutdown

This is a feature of xl only.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
9 years agots-kernel-build: enable PVH support
Roger Pau Monne [Thu, 28 May 2015 16:00:57 +0000 (17:00 +0100)]
ts-kernel-build: enable PVH support

Enable PVH support on kernels build by OSSTest.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoFreeBSD: use custom image containing atkbd fix
Roger Pau Monne [Thu, 28 May 2015 15:59:08 +0000 (16:59 +0100)]
FreeBSD: use custom image containing atkbd fix

Recent failures are due to a bug in x86emu when trying to call into
the BIOS in order to get the repeat rate and keyboard frequency.  This
image contains the following fix, which should prevent the issue by
not calling into the BIOS anymore:

https://svnweb.freebsd.org/base?view=revision&revision=283219

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agogrub: remove patch to disable submenu from 20_linux_xen overlay
Ian Campbell [Wed, 6 May 2015 10:41:11 +0000 (11:41 +0100)]
grub: remove patch to disable submenu from 20_linux_xen overlay

setupboot_grub2 now supports submenus, so we can reduce our delta vs
upstream a bit.

I started by extracting 20_linux_xen from
http://snapshot.debian.org/archive/debian/20130703T094657Z/pool/main/g/grub2/grub-common_1.99-27%2Bdeb7u2_amd64.deb
and then applying the patch at
http://savannah.gnu.org/file/grub.patch?file_id=32276 (the patch from
grub bug #42420 at http://savannah.gnu.org/bugs/?43420) and
reinstating the comment at the top of the file (modified to drop the
reference to the Debian bug.

This left me with some spurious changes:

    @@ -93,7 +93,7 @@ linux_entry ()
           if test ! -e "${xen_dirname}/${xenpolicy}" ; then
              return
           fi
    -      xen_args=`echo $xen_args flask=enforcing`
    +      xen_args=`echo $xen_args flask_enabled=1 flask_enforcing=1`
           if ${recovery} ; then
              title="$(gettext_quoted "%s, with Xen %s (XSM enabled) and Linux %s (recovery mode)")"
           else
    @@ -137,7 +137,6 @@ EOF
            echo    '$message'
            module  ${rel_dirname}/${xenpolicy}
     EOF
    -  fi
       cat << EOF
     }
     EOF

I think these are bugs in the patch in the grub BTS, which were fixed
while iterating over the XSM series in osstest but didn't make it into
the upstream version, the fixes to those bugs are reverted byu the
above. So I have manually reverted them.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei.Lui@citrix.com
Cc: longtaox.pang@intel.com
9 years agoChanges to support '/boot' leading paths of kernel, xen, in grub
longtao.pang [Tue, 26 May 2015 09:08:20 +0000 (17:08 +0800)]
Changes to support '/boot' leading paths of kernel, xen, in grub

Support situations of grub that have vmlinuz and other things starting
with path of '/boot' rather than '/'.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoParsing grub which has 'submenu' primitive
longtao.pang [Tue, 26 May 2015 09:08:19 +0000 (17:08 +0800)]
Parsing grub which has 'submenu' primitive

Now auto-gen kernel grub2 config file's boot menu entries can have
2-level hierarchy, containing 'submenu' primitive, which is comprised by
several sub-menuentries. Xen boot entries are grouped into such kind of
'submenu' block. This patch adds setupboot_grub2() ability to handle
such new grub.cfg format

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
[ ijc -- minor coding style fixup and dropped comment made obsolete by
         this change ]

9 years agocr-for-branches: Add linux-3.18
Ian Campbell [Tue, 26 May 2015 11:44:09 +0000 (12:44 +0100)]
cr-for-branches: Add linux-3.18

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoMove the code for setting memory size into prep()
longtao.pang [Wed, 13 May 2015 03:36:40 +0000 (11:36 +0800)]
Move the code for setting memory size into prep()

Move the code for setting memory size from outside into prep(),
where these $ram_mb operations are only used; And in next patch,
for L2 guest installation, $ram_mb will be overridden by a runvar value,
which needs $gho in scope.

This is pure code motion. No functional change.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoEdit some APIs in TestSupport.pm for nested test
longtao.pang [Wed, 13 May 2015 03:36:39 +0000 (11:36 +0800)]
Edit some APIs in TestSupport.pm for nested test

1. If vif model is defined for guest, use it in guest's configuration
2. In L2 installation context, its host (L1) shall not have a static IP
but stored in runvar in L1 installation before; use this stored runvar
IP.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
[ ijc -- resolved conflict with addition of ${viftype}, made
         ${vifmodel} a prefix rather than postfix thing for
         consistency with ${viftype}. ]

9 years agoRefactor installation of overlays
longtao.pang [Wed, 13 May 2015 03:36:38 +0000 (11:36 +0800)]
Refactor installation of overlays

Apply overlay to guests as well as hosts by moving to preseed_base.

We do this because we want to be able to:
       * Use '/etc/init.d/osstest-confirm-booted' for guests as
         well as hosts (in particular when testing nestedhvm
         we sometimes want to treat a guest as a host).
       * Use grub's 20_linux_xen which is harmless unless Xen is
         installed in the guest which is the case only for
         nestedhvm testing when we want these changes.
The other things in the overlay are some initscripts
(xenbridge and xenlightdaemons) which are not enabled for guests and
hence are therefore harmless.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoRevert "cs-bisection-step: allow -bisect blessed flights for basis pass"
Ian Jackson [Wed, 20 May 2015 12:43:39 +0000 (13:43 +0100)]
Revert "cs-bisection-step: allow -bisect blessed flights for basis pass"

If the bisector is allowed to consider its own output flights as
candidates for the basis pass, then it will (generally) restart with a
new basis each time it finds a pass.

This is very slow and wasteful.

There is not much explanation in 01edca47 of the change I am now
reverting, but I think I probably created by some semi-manual process
a flight to serve as the basis.  I now think that my mistake was to
bless that flight `adhoc-bisect' or some such, rather than `adhoc'.

This reverts commit 01edca47be3742a1660b1956c1f06ca934b97352.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocs-bisection-step: Do not treat repro attempts as flail
Ian Jackson [Wed, 20 May 2015 12:36:15 +0000 (13:36 +0100)]
cs-bisection-step: Do not treat repro attempts as flail

The need_repro machinery deliberarely makes attempts to reproduce
various results.

This can cause the flail detector to trigger when not intended.  In
particular, the bisector may have (for some reason[1]) restarted with
a new baseline, and the temporarally-stripy pass/fail requirement
would then require the basis fail to be repro'd, again.

[1] Currently this happens much more often than is desirable.  This
will be fixed in a moment.

Fix this by only considering, for the purposes of flail, flights which
are no older than the first repro (the basis pass).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoRevert "cs-bisection-step: Abandon repro attempts after a bit"
Ian Jackson [Wed, 20 May 2015 11:30:19 +0000 (12:30 +0100)]
Revert "cs-bisection-step: Abandon repro attempts after a bit"

This safety catch is unnecessary and unhelpful.

It is unnecessary because 489773b4 "Detect flailing" will detect
attempts by the bisector to repeatedly run the same flight and hope
for different results.

It is unhelpful because it can happen for good reasons that a
particular revision has been tested many times.  In particular:

 - The osstest push gate input tree may have not been advanced for a
   long time and been failing its push gate.

 - The bisector may have (for some reason[1]) restarted with a new
   baseline, and the temporarally-stripy pass/fail requirement would
   then require the basis fail to be repro'd, again.

[1] Currently this happens much more often than is desirable.  This
will be fixed in a moment.

This reverts commit 2676277181599a889657354028b992379aa6142b.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocs-bisection-step: Flail detection: look only at our blessing
Ian Jackson [Wed, 20 May 2015 12:11:41 +0000 (13:11 +0100)]
cs-bisection-step: Flail detection: look only at our blessing

When looking for identical previously flights, consider only ones
which have the same blessing as our prospective flight will have.

There are good reasons why apparently identical flights might appear
with other in-scope blessings: notably, a single-test-job main branch
might produce many failures from its push gate, which would all have
the main branch blessing (rather than the bisector's blessing).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocs-bisection-step: Clarify and correct flailing message
Ian Jackson [Wed, 20 May 2015 12:06:37 +0000 (13:06 +0100)]
cs-bisection-step: Clarify and correct flailing message

There are many other possible reasons for this, besides a bug in the
build version machinery.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocs-bisection-step: Refer to jobs we create just by job name
Ian Jackson [Wed, 20 May 2015 11:49:05 +0000 (12:49 +0100)]
cs-bisection-step: Refer to jobs we create just by job name

When we make a fresh build job, rather than referring to an existing
job in another flight, pass to the rest of the machinery only the job
name, not <flight>.<job>.

This means that the generated flight refers to its own jobs without
specifying the flight number.  This allows the flail detector to
operate properly: without this, we might have repeated attempts to
test and build the same thing, but they would look identical because
their self-referential runvars would be different due to their
different flight numbers.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agots-host-install: die on unknown options
Ian Jackson [Mon, 18 May 2015 12:46:07 +0000 (13:46 +0100)]
ts-host-install: die on unknown options

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agots-host-install: New --poweron-test-only option
Ian Jackson [Fri, 15 May 2015 16:54:03 +0000 (16:54 +0000)]
ts-host-install: New --poweron-test-only option

We are having a difficulty with one of the test boxes, which can be
most easily reproduced by running ts-host-install to power cycle the
box and then see if it wakes up enough to fetch a preseed file.

Keep this mode of operation in tree in case it's useful in future.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agots-host-install: Fix indentation error
Ian Jackson [Fri, 15 May 2015 16:53:33 +0000 (16:53 +0000)]
ts-host-install: Fix indentation error

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoRevert "allow.all: Ignore FreeBSD local migration failures"
Ian Jackson [Sat, 16 May 2015 00:15:30 +0000 (01:15 +0100)]
Revert "allow.all: Ignore FreeBSD local migration failures"

We think these are solved by the new FreeBSD images, in
"FreeBSD: use custom image containing BSD netfront bugfix"

This reverts commit fcb818eeb543fbf9f7d2cbc572531c0a5bf312ef.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoFreeBSD: use custom image containing BSD netfront bugfix
Ian Jackson [Fri, 15 May 2015 17:03:01 +0000 (18:03 +0100)]
FreeBSD: use custom image containing BSD netfront bugfix

Roger has supplied these images which contain a fix for a netfront bug
in FreeBSD: it would send the gratuitous ARP before netback was ready,
so it could get lost, leading to stochastic migration failures:
  https://svnweb.freebsd.org/base?view=revision&revision=282908

Here is the runvar diff:
 - test-amd64-i386-freebsd10-amd64 freebsd_image FreeBSD-10.1-RELEASE-amd64.raw.xz
 - test-amd64-i386-freebsd10-i386  freebsd_image FreeBSD-10.1-RELEASE-i386.raw.xz
 + test-amd64-i386-freebsd10-amd64 freebsd_image FreeBSD-10.1-CUSTOM-amd64-20150518.raw.xz
 + test-amd64-i386-freebsd10-i386  freebsd_image FreeBSD-10.1-CUSTOM-i386-20150518.raw.xz
I have confirmed that these are the right filenames.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agoDebian: correct u-boot command to load flask policy
Ian Campbell [Fri, 15 May 2015 18:54:36 +0000 (19:54 +0100)]
Debian: correct u-boot command to load flask policy

The use of the $flaskpolocy variable is escaped only once, meaning it
is interpreted by the shell on the test host, a context in which it
is not set (it is set in Perl, and it would be set in u-boot env by
the line above). The symptom was that the ext2load command got given
no file to load and so tries to load the default "pxelinux.0":

May 14 20:47:47.108243 ## Executing script at 43100000
May 14 20:47:47.111115 Loading dtbs/sun7i-a20-cubietruck.dtb
May 14 20:47:47.114489 22007 bytes read in 114 ms (188.5 KiB/s)
May 14 20:47:47.252237 820116 bytes read in 114 ms (6.9 MiB/s)
May 14 20:47:47.396931 Loaded xen-4.6-unstable to 0x41000000 (c8394)
May 14 20:47:47.400989 command line: [...]
May 14 20:47:47.413380 4963576 bytes read in 214 ms (22.1 MiB/s)
May 14 20:47:47.653005 Loaded vmlinuz-3.16.7-ckt4+ to 0x42000000 (4bbcf8)
May 14 20:47:47.657999 command line: [...]
May 14 20:47:47.671376 12501593 bytes read in 510 ms (23.4 MiB/s)
May 14 20:47:48.203299 Loaded initrd.img-3.16.7-ckt4+ to 0x43300000 (bec259)
May 14 20:47:48.208406 ** File not found pxelinux.0 **
May 14 20:47:48.294291 Loaded xenpolicy-4.6-unstable to 0x41200000 (bec259)

The filesize in the log message remains the same as the initrd because
nothing is actually loaded so the variable isn't updated.

Rather than adding additional backticks to quote it to be interpreted
by u-boot, remove the quoting so it is interpreted by Perl, making
the setting of the u-boot var (which in any case used the wrong
syntax) unnecessary. This matches what we do for the kernel etc too.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Wei Liu <Wei.Liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoRevert "ts-logs-capture: collect resize2fs.log"
Ian Jackson [Wed, 11 Jun 2014 17:11:23 +0000 (18:11 +0100)]
Revert "ts-logs-capture: collect resize2fs.log"

This is no longer generated.

This reverts commit 9f299b936d1812bfd958d2771676a8de3cf9c060.

9 years agots-xen-build-prep: mkfs a new /home/osstest, don't resize2fs
Ian Jackson [Wed, 11 Jun 2014 20:38:27 +0000 (21:38 +0100)]
ts-xen-build-prep: mkfs a new /home/osstest, don't resize2fs

Online resize is 40x slower than mkfs.  It appears that the
backgrounded resize2fs can starve build tasks of IO bandwidth.

So instead, use mkfs to make a new filesystem for /home/osstest.
We use rsync to copy in the old contents.

For convenience of (a) review (b) possible reversion, we keep (for
now) the lvextend machinery.  So we create a new 1-extent LV for the
lvextend machinery to work on.

But we don't call resize2fs when we extend it, because now it doesn't
have a fs on it yet.  We make the filesystem later.

We move the ccache_setup until after this is done because it's a bit
pointless to put things in the to-be-removed /home/osstest when they
could be put in the new one after it had been set up.

We take slight care to make the rune slightly idempotent: if it
completed successfully we detect this and do not run it again.  But if
it didn't, things may be messed up and running it again is unlikely to
help and may make things worse.

I have tested this on rice-weevil and the whole new target command
(including rsync, mkfs, mount etc.) takes 126s.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agots-xen-build-prep: Break out determine_vg_lv
Ian Jackson [Wed, 11 Jun 2014 20:37:02 +0000 (21:37 +0100)]
ts-xen-build-prep: Break out determine_vg_lv

And make $lv a global.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoTestSupport: break out lv_dev_mapper
Ian Jackson [Wed, 11 Jun 2014 17:28:55 +0000 (18:28 +0100)]
TestSupport: break out lv_dev_mapper

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
9 years agots-xen-build-prep: Abolish "background" parameter to lvextend
Ian Jackson [Wed, 11 Jun 2014 17:13:10 +0000 (18:13 +0100)]
ts-xen-build-prep: Abolish "background" parameter to lvextend

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
9 years agoRevert "ts-xen-build-prep: Run the backgrounded resize2fs with ionice"
Ian Jackson [Wed, 11 Jun 2014 17:10:43 +0000 (18:10 +0100)]
Revert "ts-xen-build-prep: Run the backgrounded resize2fs with ionice"

This did not really help.  We are going to do something different.

This reverts commit ee31a32af1fc9a38ad08b4a36ee69e1c58ea0828.

9 years agoDebian: Log which kernel/initrd we failed to copy.
Ian Campbell [Fri, 6 Mar 2015 10:05:34 +0000 (10:05 +0000)]
Debian: Log which kernel/initrd we failed to copy.

Helpful to trackdown which config var I've got wrong.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoi18n/l10n: Make Timezone configurable and change the default
Ian Jackson [Fri, 15 May 2015 10:31:35 +0000 (11:31 +0100)]
i18n/l10n: Make Timezone configurable and change the default

* Introduce a new config option Timezone
* Replace hardcoded Europe/London everywhere with $c{Timezone}
* The default is UTC
* But in production-config-cambridge set it to Europe/London

The overall effect is:
* No change in Cambridge
* Default timezone changes to UTC but can now be overridden
* Production instance timezone changes to UTC

(It appears that there is no reasonable way to find out the Olson TZ
name of the controller host's default timezone.  If there were, or we
discover one, we should arrange that the default is set
appropriately.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agots-logs-capture: Collect /var/log/messages and /var/log/debug
Ian Jackson [Thu, 14 May 2015 17:41:21 +0000 (18:41 +0100)]
ts-logs-capture: Collect /var/log/messages and /var/log/debug

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoproduction-config: Use /home/logs, not /home/osstest/pub
Ian Jackson [Thu, 14 May 2015 17:12:30 +0000 (18:12 +0100)]
production-config: Use /home/logs, not /home/osstest/pub

The logs and images (including .../logs, .../results, etc.) are now on
their own filesystem on the production osstest VM, which I have called
/home/logs.

Changing this in production config will allow us to tidy up by
removing the symlink I left behind.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agosg-run-job: Repeat rumpuserxen-demo-xenstorels test 150 times
Ian Campbell [Fri, 17 Apr 2015 10:01:29 +0000 (11:01 +0100)]
sg-run-job: Repeat rumpuserxen-demo-xenstorels test 150 times

With 50 we are still seeing occasional spurious passes.

Looking at some logs the current 50 iterations takes around 4 minutes
to run (3-4s per iteration).

I think we can justify spending 10 minutes running this test, which is
150x4s.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomake-flight: Drop pointless jobs from seabios and ovmf flights
Ian Campbell [Tue, 12 May 2015 08:10:33 +0000 (09:10 +0100)]
make-flight: Drop pointless jobs from seabios and ovmf flights

These are both firmware which is used only with qemuu tests. There is
no point testing either PV guets or HVM with qemut (which always uses
rombios).

Comparing before and after of:

./standalone-generate-dump-flight-runvars | \
    awk '// {printf "%-32s %s\n", $1, $2}' | \
   sort -u

Results in:

    @@ -615,48 +615,19 @@
     ovmf                             build-i386-libvirt
     ovmf                             build-i386-pvops
     ovmf                             build-i386-xsm
    -ovmf                             test-amd64-amd64-libvirt
    -ovmf                             test-amd64-amd64-libvirt-xsm
    -ovmf                             test-amd64-amd64-pair
    -ovmf                             test-amd64-amd64-xl
    -ovmf                             test-amd64-amd64-xl-credit2
    -ovmf                             test-amd64-amd64-xl-multivcpu
    -ovmf                             test-amd64-amd64-xl-pvh-amd
    -ovmf                             test-amd64-amd64-xl-pvh-intel
    -ovmf                             test-amd64-amd64-xl-qemut-debianhvm-amd64
    -ovmf                             test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm
    -ovmf                             test-amd64-amd64-xl-qemut-win7-amd64
    -ovmf                             test-amd64-amd64-xl-qemut-winxpsp3
     ovmf                             test-amd64-amd64-xl-qemuu-debianhvm-amd64
     ovmf                             test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm
     ovmf                             test-amd64-amd64-xl-qemuu-ovmf-amd64
     ovmf                             test-amd64-amd64-xl-qemuu-win7-amd64
     ovmf                             test-amd64-amd64-xl-qemuu-winxpsp3
    -ovmf                             test-amd64-amd64-xl-sedf
    -ovmf                             test-amd64-amd64-xl-sedf-pin
    -ovmf                             test-amd64-amd64-xl-xsm
    -ovmf                             test-amd64-i386-freebsd10-amd64
    -ovmf                             test-amd64-i386-freebsd10-i386
    -ovmf                             test-amd64-i386-libvirt
    -ovmf                             test-amd64-i386-libvirt-xsm
    -ovmf                             test-amd64-i386-pair
    -ovmf                             test-amd64-i386-qemut-rhel6hvm-amd
    -ovmf                             test-amd64-i386-qemut-rhel6hvm-intel
     ovmf                             test-amd64-i386-qemuu-rhel6hvm-amd
     ovmf                             test-amd64-i386-qemuu-rhel6hvm-intel
    -ovmf                             test-amd64-i386-xl
    -ovmf                             test-amd64-i386-xl-qemut-debianhvm-amd64
    -ovmf                             test-amd64-i386-xl-qemut-debianhvm-amd64-xsm
    -ovmf                             test-amd64-i386-xl-qemut-win7-amd64
    -ovmf                             test-amd64-i386-xl-qemut-winxpsp3
    -ovmf                             test-amd64-i386-xl-qemut-winxpsp3-vcpus1
     ovmf                             test-amd64-i386-xl-qemuu-debianhvm-amd64
     ovmf                             test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm
     ovmf                             test-amd64-i386-xl-qemuu-ovmf-amd64
     ovmf                             test-amd64-i386-xl-qemuu-win7-amd64
     ovmf                             test-amd64-i386-xl-qemuu-winxpsp3
     ovmf                             test-amd64-i386-xl-qemuu-winxpsp3-vcpus1
    -ovmf                             test-amd64-i386-xl-xsm
     qemu-mainline                    build-amd64
     qemu-mainline                    build-amd64-libvirt
     qemu-mainline                    build-amd64-pvops
    @@ -906,48 +877,19 @@
     seabios                          build-i386-libvirt
     seabios                          build-i386-pvops
     seabios                          build-i386-xsm
    -seabios                          test-amd64-amd64-libvirt
    -seabios                          test-amd64-amd64-libvirt-xsm
    -seabios                          test-amd64-amd64-pair
    -seabios                          test-amd64-amd64-xl
    -seabios                          test-amd64-amd64-xl-credit2
    -seabios                          test-amd64-amd64-xl-multivcpu
    -seabios                          test-amd64-amd64-xl-pvh-amd
    -seabios                          test-amd64-amd64-xl-pvh-intel
    -seabios                          test-amd64-amd64-xl-qemut-debianhvm-amd64
    -seabios                          test-amd64-amd64-xl-qemut-debianhvm-amd64-xsm
    -seabios                          test-amd64-amd64-xl-qemut-win7-amd64
    -seabios                          test-amd64-amd64-xl-qemut-winxpsp3
     seabios                          test-amd64-amd64-xl-qemuu-debianhvm-amd64
     seabios                          test-amd64-amd64-xl-qemuu-debianhvm-amd64-xsm
     seabios                          test-amd64-amd64-xl-qemuu-ovmf-amd64
     seabios                          test-amd64-amd64-xl-qemuu-win7-amd64
     seabios                          test-amd64-amd64-xl-qemuu-winxpsp3
    -seabios                          test-amd64-amd64-xl-sedf
    -seabios                          test-amd64-amd64-xl-sedf-pin
    -seabios                          test-amd64-amd64-xl-xsm
    -seabios                          test-amd64-i386-freebsd10-amd64
    -seabios                          test-amd64-i386-freebsd10-i386
    -seabios                          test-amd64-i386-libvirt
    -seabios                          test-amd64-i386-libvirt-xsm
    -seabios                          test-amd64-i386-pair
    -seabios                          test-amd64-i386-qemut-rhel6hvm-amd
    -seabios                          test-amd64-i386-qemut-rhel6hvm-intel
     seabios                          test-amd64-i386-qemuu-rhel6hvm-amd
     seabios                          test-amd64-i386-qemuu-rhel6hvm-intel
    -seabios                          test-amd64-i386-xl
    -seabios                          test-amd64-i386-xl-qemut-debianhvm-amd64
    -seabios                          test-amd64-i386-xl-qemut-debianhvm-amd64-xsm
    -seabios                          test-amd64-i386-xl-qemut-win7-amd64
    -seabios                          test-amd64-i386-xl-qemut-winxpsp3
    -seabios                          test-amd64-i386-xl-qemut-winxpsp3-vcpus1
     seabios                          test-amd64-i386-xl-qemuu-debianhvm-amd64
     seabios                          test-amd64-i386-xl-qemuu-debianhvm-amd64-xsm
     seabios                          test-amd64-i386-xl-qemuu-ovmf-amd64
     seabios                          test-amd64-i386-xl-qemuu-win7-amd64
     seabios                          test-amd64-i386-xl-qemuu-winxpsp3
     seabios                          test-amd64-i386-xl-qemuu-winxpsp3-vcpus1
    -seabios                          test-amd64-i386-xl-xsm
     xen-4.0-testing                  build-amd64
     xen-4.0-testing                  build-amd64-libvirt
     xen-4.0-testing                  build-amd64-pvops

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoDebian: Apply uboot size_hex_prefix workaround to XSM policy too
Ian Campbell [Mon, 4 May 2015 08:18:54 +0000 (09:18 +0100)]
Debian: Apply uboot size_hex_prefix workaround to XSM policy too

Worksaround "Sorry, I could not convert "be700c>"" messages.

The big comment and code which determines $size_hex_prefix is moved
unchanged.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agots-xen-*: Avoid use of uninitialised $r{"enable_xsm"}
Ian Campbell [Thu, 2 Apr 2015 07:33:40 +0000 (08:33 +0100)]
ts-xen-*: Avoid use of uninitialised $r{"enable_xsm"}

Use of uninitialized value $r{"enable_xsm"} in pattern match (m//) at ./ts-xen-install line 49.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoToolstack::libvirt: Support for ACPI fallback for shutdown
Ian Campbell [Thu, 7 May 2015 15:37:18 +0000 (16:37 +0100)]
Toolstack::libvirt: Support for ACPI fallback for shutdown

This is the libvirt counterpart to "Toolstack::xl: Support for ACPI
fallback for shutdown". Currently there are no jobs which test HVM
guests with libvirt and so this is completely untested in the context
of osstest (but at least should be harmless to current jobs).

This relies on an assumption that "virsh shutdown" behaves the same as
"virsh reboot" and accepts a comma separated list of methods to try
given to the --mode argument, which Jim has tested and confirmed to be
true.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Jim Fehlig <jfehlig@suse.com>
Acked by: Jim Fehlig <jfehlig@suse.com>

9 years agoToolstack::xl: Support for ACPI fallback for shutdown
Ian Campbell [Thu, 7 May 2015 09:07:04 +0000 (10:07 +0100)]
Toolstack::xl: Support for ACPI fallback for shutdown

HVM guests which do not include PV drivers will not shutdown after a
simple "xl shutdown". Add a runvar to indicate that the guest will
shutdown in response to an ACPI power event and apply this to the win7
and winxp test jobs.

Tested with:
    test-amd64-amd64-xl-qemuu-winxpsp3
    test-amd64-amd64-xl-qemuu-win7-amd64
    test-amd64-amd64-xl-qemuu-debianhvm-amd64
    test-amd64-amd64-xl

The first two correctly used the extra flag and the domains shutdown,
the other two correctly didn't use the extra flag and still shutdown.

Not currently implemented for libvirt, since we currently have no
libvirt tests I could try it with.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomake-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()
Ian Campbell [Wed, 6 May 2015 09:56:38 +0000 (10:56 +0100)]
make-flight: Handle $BUILD_LVEXTEND_MAX in mfi-common:create_build_jobs()

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agots-debian-hvm-install: Only apply EFI workaround for Wheezy
Ian Campbell [Wed, 6 May 2015 09:56:37 +0000 (10:56 +0100)]
ts-debian-hvm-install: Only apply EFI workaround for Wheezy

The previous refactoring of preseed hooks makes this easy to do.

The underlying issue is lack of persistent variable store in our OVMF
setup, which we workaround by placing a copy of grub at the removable
media path. Add a comment saying this since I initially thought this
was just a Wheezy bug.

In Jessie the extra copy of grub can be achieved by preseeding
grub-installer/force-efi-extra-removable (since various real h/w has
similar limitations/bugs) however I haven't tested that so I didn't
add it to the preseed yet, I just mention it in the code comment.

Currently this script hardcodes Wheezy, refactor to use the 'suite'
guest_var (or $c{GuestDebianSuite})

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: wei.liu2@citrix.com
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoDebian: add preseed_create_guest helper
Ian Campbell [Fri, 25 Apr 2014 20:29:24 +0000 (21:29 +0100)]
Debian: add preseed_create_guest helper

Creates a preseed file suitable for use in a PV guest

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoDebian: Refactor installation of overlays, so it can be used for guests too
Ian Campbell [Wed, 6 May 2015 09:56:35 +0000 (10:56 +0100)]
Debian: Refactor installation of overlays, so it can be used for guests too

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: longtaox.pang@intel.com
9 years agoDebian: refactor preseeding of .ssh directories
Ian Campbell [Wed, 6 May 2015 09:56:34 +0000 (10:56 +0100)]
Debian: refactor preseeding of .ssh directories

Causes known_hosts to be consistently created as well as ~osstest/.ssh
to be consistently populated (it previsouly wasn't for HVM guests).

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: longtaox.pang@intel.com
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoDebian: refactor code to add preseed commands to the preseed file
Ian Campbell [Wed, 6 May 2015 09:56:33 +0000 (10:56 +0100)]
Debian: refactor code to add preseed commands to the preseed file

Call it from ts-debian-hvm-install.

This means that, in future, ts-debian-di-install can use
preseed_hook_command and preseed_hook_installscript.

The existing opencoded use of d-i/late_command in the guest preseed
needs to become a preseed_hook_command so as not to clash with the use
of preseed_hook_cmds().

This requires also adding a #! line and the "set -ex" boilerplate
which in turn requires slightly rewriting the /boot/EFI handling part
to also work if the system is not installed for EFI (in which case
grubx64.efi isn't installed). Previously this would have needlessly
created the directory and then ignored the error from cp.

The ssh authorized keys bit isn't touched since it works as is and
will go away in a subsequent patch.

Apart from no longer creating /target/boot/efi/EFI/boot when it is not
needed there is no functional change for now.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: longtaox.pang@intel.com
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agocreate_webfile: Support use with guests as well as hosts.
Ian Campbell [Wed, 6 May 2015 09:56:32 +0000 (10:56 +0100)]
create_webfile: Support use with guests as well as hosts.

In particular make the path unique by ensuring it includes the host
and guest name in the guest case.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoTestSupport: allow caller of prepareguest_part_xencfg to specify viftype
Ian Campbell [Wed, 6 May 2015 09:56:31 +0000 (10:56 +0100)]
TestSupport: allow caller of prepareguest_part_xencfg to specify viftype

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoMerge remote-tracking branch 'cam/production'
Ian Jackson [Tue, 12 May 2015 12:36:27 +0000 (13:36 +0100)]
Merge remote-tracking branch 'cam/production'

9 years agoDocs (in new Docs directory) for cr-disk-report
Ian Jackson [Tue, 12 May 2015 10:41:12 +0000 (11:41 +0100)]
Docs (in new Docs directory) for cr-disk-report

The PDF is automatically generated and linked to, when applicable.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agocr-disk-report: New script
Ian Jackson [Mon, 11 May 2015 15:49:18 +0000 (16:49 +0100)]
cr-disk-report: New script

This generates a report on the disk usage in the logs directory.

9 years agoLogs: Break out logs_select etc. into new Osstest::Management
Ian Jackson [Mon, 11 May 2015 14:24:50 +0000 (15:24 +0100)]
Logs: Break out logs_select etc. into new Osstest::Management

We are going to want to reuse these minor bits of
cr-ensure-disk-space.  Break them out into a new perl module.

We also need to rename some things to make them have names more
suitable for a wider namespace, even if only selectively exported:
 * @logsshopts from @sshopts (it is not the same variable as
   Osstest::TestSupport::sshopts).
 * $loghost and $logdir from $pubhost and $pubdir).
 * onloghost from ontarget.
 * logcfg from dircfg.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agocs-bisection-step: Detect flailing
Ian Jackson [Mon, 11 May 2015 13:30:34 +0000 (14:30 +0100)]
cs-bisection-step: Detect flailing

Specifically, search for previous identical flights (on the same
branch with the relevant blessings).  Here `identical' means it has
exactly the same set of runvars (and, hence, the same set of jobs,
assuming each job has at least one runvar).

This detects various forms of looping which aren't stopped any other
way.

Specifically, one relevant situation occurs if attempts to build
revision (A,B,C) actually build (A,B,C') due to bugs in the build
machinery (which could be bugs in osstest or in the software under
test).  In this case the bisector would never spot its previous
attempts as relevant; instead, it would disregard them due to the
mismatched versions.  It would then retry the same version, until
something happened to stop it.

As written here, we do not consider osstest revision as a relevant
factor in `identical'.  So if reason for the looping is a bug in
osstest we would need to manually un-bless affected flights, as well
as removing the stamp files which are used to record completion of
attempted bisection.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoREADME.planner: Document the resource planning system
Ian Jackson [Thu, 20 Nov 2014 18:07:16 +0000 (18:07 +0000)]
README.planner: Document the resource planning system

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agoproduction-config: Correct syntax for clearing OsstestUpstream.
Ian Campbell [Mon, 11 May 2015 10:37:33 +0000 (11:37 +0100)]
production-config: Correct syntax for clearing OsstestUpstream.

Previous syntax actually sets it to the literal string containing two
quotation marks.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agoallow.all: Ignore FreeBSD local migration failures
Ian Campbell [Mon, 11 May 2015 10:35:56 +0000 (11:35 +0100)]
allow.all: Ignore FreeBSD local migration failures

Until the issues with them are resolved

Rerunning sg-report-flight for 54309 results in:

@@ -5,11 +9,11 @@

 Tests which did not succeed and are blocking,
 including tests which could not be run:
- test-amd64-i386-freebsd10-amd64 13 guest-localmigrate     fail REGR. vs. 50405
  build-amd64-xsm               5 xen-build                 fail REGR. vs. 50405

 Regressions which are regarded as allowable (not blocking):
  test-armhf-armhf-xl-sedf-pin  3 host-install(3)         broken REGR. vs. 50405
+ test-amd64-i386-freebsd10-amd64 13 guest-localmigrate     fail REGR. vs. 50405
  test-amd64-i386-freebsd10-i386 13 guest-localmigrate           fail like 50405
  test-armhf-armhf-libvirt     11 guest-start                  fail   like 50405

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agoAutomerge of git://xenbits.xen.org/osstest.git master into production
xen.org [Sun, 10 May 2015 16:35:00 +0000 (17:35 +0100)]
Automerge of git://xenbits.xen.org/osstest.git master into production

10 years agocambridge: Reduce reliance on woking and configdb
Ian Campbell [Thu, 7 May 2015 15:41:21 +0000 (16:41 +0100)]
cambridge: Reduce reliance on woking and configdb

The hosts which were previsouly attached to woking's serial ports are
now attached to osstser1 which is our own box.

All hosts which previously used configdb (and by extension statedb)
are now configured directly via the host properties entries for PDU
control (direct to PDU) and for Ethernet address, so disable configdb.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
10 years agoMerge branch 'from-cambridge/2015-05-05' of xenbits.xen.org:/home/xen/git/people...
Ian Jackson [Thu, 7 May 2015 14:06:59 +0000 (15:06 +0100)]
Merge branch 'from-cambridge/2015-05-05' of xenbits.xen.org:/home/xen/git/people/ianc/osstest into fast