Ian Jackson [Fri, 29 May 2015 14:14:41 +0000 (14:14 +0000)]
sg-report-job-history: Cope if history too short
If there have been less than 99 relevant flights, the inner SELECT (to
determine the minimum flight number) would return NULL. And anything
> NULL is NULL and NULL is treated as false. So the host runvar
identification would break.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Mon, 15 Jun 2015 08:36:02 +0000 (09:36 +0100)]
Arrange to upgrade microcode on x86 test hosts.
Both Xen and Linux support extracting a microcode update from an
initramfs early during boot. This requires prepending a suitable
uncompressed cpio archive containing the necessary files to the
initrd.
Xen also supports loading the microcode cpio from any multiboot
module, but for in order to allow the possibility of loading on native
boots (e.g. for build jobs) we prefer the prepend method.
This patch provides mg-cpu-microcode-update which creates a suitable
microcode cpio in the images directory and arranges for it to be added
to the host during installation (so it is done before the kernel is
installed and initramfs generated etc, saving faff). It also adds
"ucode=scan" to the Xen command line when necessary.
The version of initramfs-tools in Wheezy does not yet support
prepending things to the initrm, so we use a custom compression
command which sneakily does it for us. This could be done better from
Jessie onwards.
Note that Linux only supports this from v3.8 onwards, so this doesn't
work for Wheezy (which uses v3.2). From Jessie onwards we should
benefit from microcode updates even for native (build) jobs.
Tested on both Intel and AMD where it appeared to have the desired
effect under Xen. Under native I only tried Wheezy which doesn't
support early microcode.
Note that I've not bothered to implement late microcode updates, which
would work on v3.2 too, since that would need a different set of files
etc and it doesn't seem especially pressing.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Cc: Jan Beulich <JBeulich@suse.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Roger Pau Monne [Fri, 29 May 2015 10:38:18 +0000 (11:38 +0100)]
osstest: reduce FreeBSD install timeouts
Only the first block is expected to take longer (because it decompresses
the image and writes it to a LVM volume), the remaining commands should
execute much faster, so reduce the timeout.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 26 May 2015 11:36:00 +0000 (12:36 +0100)]
ts-libvirt-build: initscript should depend on xenstored service
In order that xencommons is guarenteed to have been started before
libvirtd. Otherwise sometimes libvirt can be started first resulting
in:
error: invalid argument: unsupported config type xen-xl
Because xen wasn't available when libvirt started.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 9 Jun 2015 09:54:00 +0000 (10:54 +0100)]
Ignore guest-stop failures for windows 7 tests.
As described in <1432284841.10746.136.camel@citrix.com> /
http://lists.xen.org/archives/html/xen-devel/2015-05/msg03016.html
Windows 7 does not appear to reliably actually shutdown when asked to
via the ACPI power button.
This test step has been failing forever until the "Toolstack::xl:
Support for ACPI fallback for shutdown" patch to osstest, and now it
passes only very infrequently, so we may as well ignore it.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Cc: Andrew Cooper <Andrew.Cooper3@citrix.com> Cc: Paul Durrant <paul.durrant@citrix.com> Cc: Ian Jackson <Ian.Jackson@eu.citrix.com>, Cc: Jan Beulich <JBeulich@suse.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Roger Pau Monne [Thu, 28 May 2015 16:00:57 +0000 (17:00 +0100)]
ts-kernel-build: enable PVH support
Enable PVH support on kernels build by OSSTest.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Roger Pau Monne [Thu, 28 May 2015 15:59:08 +0000 (16:59 +0100)]
FreeBSD: use custom image containing atkbd fix
Recent failures are due to a bug in x86emu when trying to call into
the BIOS in order to get the repeat rate and keyboard frequency. This
image contains the following fix, which should prevent the issue by
not calling into the BIOS anymore:
Ian Campbell [Wed, 6 May 2015 10:41:11 +0000 (11:41 +0100)]
grub: remove patch to disable submenu from 20_linux_xen overlay
setupboot_grub2 now supports submenus, so we can reduce our delta vs
upstream a bit.
I started by extracting 20_linux_xen from
http://snapshot.debian.org/archive/debian/20130703T094657Z/pool/main/g/grub2/grub-common_1.99-27%2Bdeb7u2_amd64.deb
and then applying the patch at
http://savannah.gnu.org/file/grub.patch?file_id=32276 (the patch from
grub bug #42420 at http://savannah.gnu.org/bugs/?43420) and
reinstating the comment at the top of the file (modified to drop the
reference to the Debian bug.
This left me with some spurious changes:
@@ -93,7 +93,7 @@ linux_entry ()
if test ! -e "${xen_dirname}/${xenpolicy}" ; then
return
fi
- xen_args=`echo $xen_args flask=enforcing`
+ xen_args=`echo $xen_args flask_enabled=1 flask_enforcing=1`
if ${recovery} ; then
title="$(gettext_quoted "%s, with Xen %s (XSM enabled) and Linux %s (recovery mode)")"
else
@@ -137,7 +137,6 @@ EOF
echo '$message'
module ${rel_dirname}/${xenpolicy}
EOF
- fi
cat << EOF
}
EOF
I think these are bugs in the patch in the grub BTS, which were fixed
while iterating over the XSM series in osstest but didn't make it into
the upstream version, the fixes to those bugs are reverted byu the
above. So I have manually reverted them.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> Cc: Wei.Lui@citrix.com Cc: longtaox.pang@intel.com
longtao.pang [Tue, 26 May 2015 09:08:19 +0000 (17:08 +0800)]
Parsing grub which has 'submenu' primitive
Now auto-gen kernel grub2 config file's boot menu entries can have
2-level hierarchy, containing 'submenu' primitive, which is comprised by
several sub-menuentries. Xen boot entries are grouped into such kind of
'submenu' block. This patch adds setupboot_grub2() ability to handle
such new grub.cfg format
Signed-off-by: longtao.pang <longtaox.pang@intel.com> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
[ ijc -- minor coding style fixup and dropped comment made obsolete by
this change ]
longtao.pang [Wed, 13 May 2015 03:36:40 +0000 (11:36 +0800)]
Move the code for setting memory size into prep()
Move the code for setting memory size from outside into prep(),
where these $ram_mb operations are only used; And in next patch,
for L2 guest installation, $ram_mb will be overridden by a runvar value,
which needs $gho in scope.
This is pure code motion. No functional change.
Signed-off-by: longtao.pang <longtaox.pang@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
longtao.pang [Wed, 13 May 2015 03:36:39 +0000 (11:36 +0800)]
Edit some APIs in TestSupport.pm for nested test
1. If vif model is defined for guest, use it in guest's configuration
2. In L2 installation context, its host (L1) shall not have a static IP
but stored in runvar in L1 installation before; use this stored runvar
IP.
Signed-off-by: longtao.pang <longtaox.pang@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
[ ijc -- resolved conflict with addition of ${viftype}, made
${vifmodel} a prefix rather than postfix thing for
consistency with ${viftype}. ]
longtao.pang [Wed, 13 May 2015 03:36:38 +0000 (11:36 +0800)]
Refactor installation of overlays
Apply overlay to guests as well as hosts by moving to preseed_base.
We do this because we want to be able to:
* Use '/etc/init.d/osstest-confirm-booted' for guests as
well as hosts (in particular when testing nestedhvm
we sometimes want to treat a guest as a host).
* Use grub's 20_linux_xen which is harmless unless Xen is
installed in the guest which is the case only for
nestedhvm testing when we want these changes.
The other things in the overlay are some initscripts
(xenbridge and xenlightdaemons) which are not enabled for guests and
hence are therefore harmless.
Signed-off-by: longtao.pang <longtaox.pang@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 20 May 2015 12:43:39 +0000 (13:43 +0100)]
Revert "cs-bisection-step: allow -bisect blessed flights for basis pass"
If the bisector is allowed to consider its own output flights as
candidates for the basis pass, then it will (generally) restart with a
new basis each time it finds a pass.
This is very slow and wasteful.
There is not much explanation in 01edca47 of the change I am now
reverting, but I think I probably created by some semi-manual process
a flight to serve as the basis. I now think that my mistake was to
bless that flight `adhoc-bisect' or some such, rather than `adhoc'.
Ian Jackson [Wed, 20 May 2015 12:36:15 +0000 (13:36 +0100)]
cs-bisection-step: Do not treat repro attempts as flail
The need_repro machinery deliberarely makes attempts to reproduce
various results.
This can cause the flail detector to trigger when not intended. In
particular, the bisector may have (for some reason[1]) restarted with
a new baseline, and the temporarally-stripy pass/fail requirement
would then require the basis fail to be repro'd, again.
[1] Currently this happens much more often than is desirable. This
will be fixed in a moment.
Fix this by only considering, for the purposes of flail, flights which
are no older than the first repro (the basis pass).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 20 May 2015 11:30:19 +0000 (12:30 +0100)]
Revert "cs-bisection-step: Abandon repro attempts after a bit"
This safety catch is unnecessary and unhelpful.
It is unnecessary because 489773b4 "Detect flailing" will detect
attempts by the bisector to repeatedly run the same flight and hope
for different results.
It is unhelpful because it can happen for good reasons that a
particular revision has been tested many times. In particular:
- The osstest push gate input tree may have not been advanced for a
long time and been failing its push gate.
- The bisector may have (for some reason[1]) restarted with a new
baseline, and the temporarally-stripy pass/fail requirement would
then require the basis fail to be repro'd, again.
[1] Currently this happens much more often than is desirable. This
will be fixed in a moment.
Ian Jackson [Wed, 20 May 2015 12:11:41 +0000 (13:11 +0100)]
cs-bisection-step: Flail detection: look only at our blessing
When looking for identical previously flights, consider only ones
which have the same blessing as our prospective flight will have.
There are good reasons why apparently identical flights might appear
with other in-scope blessings: notably, a single-test-job main branch
might produce many failures from its push gate, which would all have
the main branch blessing (rather than the bisector's blessing).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 20 May 2015 11:49:05 +0000 (12:49 +0100)]
cs-bisection-step: Refer to jobs we create just by job name
When we make a fresh build job, rather than referring to an existing
job in another flight, pass to the rest of the machinery only the job
name, not <flight>.<job>.
This means that the generated flight refers to its own jobs without
specifying the flight number. This allows the flail detector to
operate properly: without this, we might have repeated attempts to
test and build the same thing, but they would look identical because
their self-referential runvars would be different due to their
different flight numbers.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 15 May 2015 16:54:03 +0000 (16:54 +0000)]
ts-host-install: New --poweron-test-only option
We are having a difficulty with one of the test boxes, which can be
most easily reproduced by running ts-host-install to power cycle the
box and then see if it wakes up enough to fetch a preseed file.
Keep this mode of operation in tree in case it's useful in future.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 15 May 2015 17:03:01 +0000 (18:03 +0100)]
FreeBSD: use custom image containing BSD netfront bugfix
Roger has supplied these images which contain a fix for a netfront bug
in FreeBSD: it would send the gratuitous ARP before netback was ready,
so it could get lost, leading to stochastic migration failures:
https://svnweb.freebsd.org/base?view=revision&revision=282908
Here is the runvar diff:
- test-amd64-i386-freebsd10-amd64 freebsd_image FreeBSD-10.1-RELEASE-amd64.raw.xz
- test-amd64-i386-freebsd10-i386 freebsd_image FreeBSD-10.1-RELEASE-i386.raw.xz
+ test-amd64-i386-freebsd10-amd64 freebsd_image FreeBSD-10.1-CUSTOM-amd64-20150518.raw.xz
+ test-amd64-i386-freebsd10-i386 freebsd_image FreeBSD-10.1-CUSTOM-i386-20150518.raw.xz
I have confirmed that these are the right filenames.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Campbell [Fri, 15 May 2015 18:54:36 +0000 (19:54 +0100)]
Debian: correct u-boot command to load flask policy
The use of the $flaskpolocy variable is escaped only once, meaning it
is interpreted by the shell on the test host, a context in which it
is not set (it is set in Perl, and it would be set in u-boot env by
the line above). The symptom was that the ext2load command got given
no file to load and so tries to load the default "pxelinux.0":
May 14 20:47:47.108243 ## Executing script at 43100000
May 14 20:47:47.111115 Loading dtbs/sun7i-a20-cubietruck.dtb
May 14 20:47:47.114489 22007 bytes read in 114 ms (188.5 KiB/s)
May 14 20:47:47.252237 820116 bytes read in 114 ms (6.9 MiB/s)
May 14 20:47:47.396931 Loaded xen-4.6-unstable to 0x41000000 (c8394)
May 14 20:47:47.400989 command line: [...]
May 14 20:47:47.413380 4963576 bytes read in 214 ms (22.1 MiB/s)
May 14 20:47:47.653005 Loaded vmlinuz-3.16.7-ckt4+ to 0x42000000 (4bbcf8)
May 14 20:47:47.657999 command line: [...]
May 14 20:47:47.671376 12501593 bytes read in 510 ms (23.4 MiB/s)
May 14 20:47:48.203299 Loaded initrd.img-3.16.7-ckt4+ to 0x43300000 (bec259)
May 14 20:47:48.208406 ** File not found pxelinux.0 **
May 14 20:47:48.294291 Loaded xenpolicy-4.6-unstable to 0x41200000 (bec259)
The filesize in the log message remains the same as the initrd because
nothing is actually loaded so the variable isn't updated.
Rather than adding additional backticks to quote it to be interpreted
by u-boot, remove the quoting so it is interpreted by Perl, making
the setting of the u-boot var (which in any case used the wrong
syntax) unnecessary. This matches what we do for the kernel etc too.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Cc: Wei Liu <Wei.Liu2@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 11 Jun 2014 20:38:27 +0000 (21:38 +0100)]
ts-xen-build-prep: mkfs a new /home/osstest, don't resize2fs
Online resize is 40x slower than mkfs. It appears that the
backgrounded resize2fs can starve build tasks of IO bandwidth.
So instead, use mkfs to make a new filesystem for /home/osstest.
We use rsync to copy in the old contents.
For convenience of (a) review (b) possible reversion, we keep (for
now) the lvextend machinery. So we create a new 1-extent LV for the
lvextend machinery to work on.
But we don't call resize2fs when we extend it, because now it doesn't
have a fs on it yet. We make the filesystem later.
We move the ccache_setup until after this is done because it's a bit
pointless to put things in the to-be-removed /home/osstest when they
could be put in the new one after it had been set up.
We take slight care to make the rune slightly idempotent: if it
completed successfully we detect this and do not run it again. But if
it didn't, things may be messed up and running it again is unlikely to
help and may make things worse.
I have tested this on rice-weevil and the whole new target command
(including rsync, mkfs, mount etc.) takes 126s.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 15 May 2015 10:31:35 +0000 (11:31 +0100)]
i18n/l10n: Make Timezone configurable and change the default
* Introduce a new config option Timezone
* Replace hardcoded Europe/London everywhere with $c{Timezone}
* The default is UTC
* But in production-config-cambridge set it to Europe/London
The overall effect is:
* No change in Cambridge
* Default timezone changes to UTC but can now be overridden
* Production instance timezone changes to UTC
(It appears that there is no reasonable way to find out the Olson TZ
name of the controller host's default timezone. If there were, or we
discover one, we should arrange that the default is set
appropriately.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 14 May 2015 17:12:30 +0000 (18:12 +0100)]
production-config: Use /home/logs, not /home/osstest/pub
The logs and images (including .../logs, .../results, etc.) are now on
their own filesystem on the production osstest VM, which I have called
/home/logs.
Changing this in production config will allow us to tidy up by
removing the symlink I left behind.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Thu, 7 May 2015 15:37:18 +0000 (16:37 +0100)]
Toolstack::libvirt: Support for ACPI fallback for shutdown
This is the libvirt counterpart to "Toolstack::xl: Support for ACPI
fallback for shutdown". Currently there are no jobs which test HVM
guests with libvirt and so this is completely untested in the context
of osstest (but at least should be harmless to current jobs).
This relies on an assumption that "virsh shutdown" behaves the same as
"virsh reboot" and accepts a comma separated list of methods to try
given to the --mode argument, which Jim has tested and confirmed to be
true.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Cc: Jim Fehlig <jfehlig@suse.com>
Acked by: Jim Fehlig <jfehlig@suse.com>
Ian Campbell [Thu, 7 May 2015 09:07:04 +0000 (10:07 +0100)]
Toolstack::xl: Support for ACPI fallback for shutdown
HVM guests which do not include PV drivers will not shutdown after a
simple "xl shutdown". Add a runvar to indicate that the guest will
shutdown in response to an ACPI power event and apply this to the win7
and winxp test jobs.
Ian Campbell [Wed, 6 May 2015 09:56:37 +0000 (10:56 +0100)]
ts-debian-hvm-install: Only apply EFI workaround for Wheezy
The previous refactoring of preseed hooks makes this easy to do.
The underlying issue is lack of persistent variable store in our OVMF
setup, which we workaround by placing a copy of grub at the removable
media path. Add a comment saying this since I initially thought this
was just a Wheezy bug.
In Jessie the extra copy of grub can be achieved by preseeding
grub-installer/force-efi-extra-removable (since various real h/w has
similar limitations/bugs) however I haven't tested that so I didn't
add it to the preseed yet, I just mention it in the code comment.
Currently this script hardcodes Wheezy, refactor to use the 'suite'
guest_var (or $c{GuestDebianSuite})
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Cc: wei.liu2@citrix.com Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 6 May 2015 09:56:33 +0000 (10:56 +0100)]
Debian: refactor code to add preseed commands to the preseed file
Call it from ts-debian-hvm-install.
This means that, in future, ts-debian-di-install can use
preseed_hook_command and preseed_hook_installscript.
The existing opencoded use of d-i/late_command in the guest preseed
needs to become a preseed_hook_command so as not to clash with the use
of preseed_hook_cmds().
This requires also adding a #! line and the "set -ex" boilerplate
which in turn requires slightly rewriting the /boot/EFI handling part
to also work if the system is not installed for EFI (in which case
grubx64.efi isn't installed). Previously this would have needlessly
created the directory and then ignored the error from cp.
The ssh authorized keys bit isn't touched since it works as is and
will go away in a subsequent patch.
Apart from no longer creating /target/boot/efi/EFI/boot when it is not
needed there is no functional change for now.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Cc: longtaox.pang@intel.com Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 11 May 2015 14:24:50 +0000 (15:24 +0100)]
Logs: Break out logs_select etc. into new Osstest::Management
We are going to want to reuse these minor bits of
cr-ensure-disk-space. Break them out into a new perl module.
We also need to rename some things to make them have names more
suitable for a wider namespace, even if only selectively exported:
* @logsshopts from @sshopts (it is not the same variable as
Osstest::TestSupport::sshopts).
* $loghost and $logdir from $pubhost and $pubdir).
* onloghost from ontarget.
* logcfg from dircfg.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Mon, 11 May 2015 13:30:34 +0000 (14:30 +0100)]
cs-bisection-step: Detect flailing
Specifically, search for previous identical flights (on the same
branch with the relevant blessings). Here `identical' means it has
exactly the same set of runvars (and, hence, the same set of jobs,
assuming each job has at least one runvar).
This detects various forms of looping which aren't stopped any other
way.
Specifically, one relevant situation occurs if attempts to build
revision (A,B,C) actually build (A,B,C') due to bugs in the build
machinery (which could be bugs in osstest or in the software under
test). In this case the bisector would never spot its previous
attempts as relevant; instead, it would disregard them due to the
mismatched versions. It would then retry the same version, until
something happened to stop it.
As written here, we do not consider osstest revision as a relevant
factor in `identical'. So if reason for the looping is a bug in
osstest we would need to manually un-bless affected flights, as well
as removing the stamp files which are used to record completion of
attempted bisection.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Mon, 11 May 2015 10:35:56 +0000 (11:35 +0100)]
allow.all: Ignore FreeBSD local migration failures
Until the issues with them are resolved
Rerunning sg-report-flight for 54309 results in:
@@ -5,11 +9,11 @@
Tests which did not succeed and are blocking,
including tests which could not be run:
- test-amd64-i386-freebsd10-amd64 13 guest-localmigrate fail REGR. vs. 50405
build-amd64-xsm 5 xen-build fail REGR. vs. 50405
Regressions which are regarded as allowable (not blocking):
test-armhf-armhf-xl-sedf-pin 3 host-install(3) broken REGR. vs. 50405
+ test-amd64-i386-freebsd10-amd64 13 guest-localmigrate fail REGR. vs. 50405
test-amd64-i386-freebsd10-i386 13 guest-localmigrate fail like 50405
test-armhf-armhf-libvirt 11 guest-start fail like 50405
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Campbell [Thu, 7 May 2015 15:41:21 +0000 (16:41 +0100)]
cambridge: Reduce reliance on woking and configdb
The hosts which were previsouly attached to woking's serial ports are
now attached to osstser1 which is our own box.
All hosts which previously used configdb (and by extension statedb)
are now configured directly via the host properties entries for PDU
control (direct to PDU) and for Ethernet address, so disable configdb.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 6 May 2015 22:47:30 +0000 (23:47 +0100)]
cs-bisection-step: Abandon repro attempts after a bit
If we have had a number of attempts at a repro, and none of them have
produced a pass or fail, something is probably wrong and we should
give up rather than carrying on.
Handle this with the machinery we use for conflicting test results.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 6 May 2015 22:41:00 +0000 (23:41 +0100)]
cs-bisection-step: Report conflict if basis pass/fail are wrong
It can happen that the (for example) supposed basis pass (originally
only tested on another host) failed (when reproduced, or for some
other reason). When that happens do not attempt to get it to pass;
instead, treat it the same way we would if we had actually got
conflicting results at that revision.
(Conversely, do not attempt to get a basis fail if the basis fail has
already passed on the selected host. This is, as it happens,
impossible in a bisection triggered by sg-report-flight with the
current invocation arrangements - but cs-bisection-step should
handle it correctly.)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 1 May 2015 14:54:03 +0000 (15:54 +0100)]
sg-report-flight: Report stepno and testid of first worst fail
This makes reading the scoreboard considerably easier.
We abuse the local variable @worst slightly, pushing the extra info we
are going to print onto the end of it.
We also have to defer printing the cells, because we compute the cell
to duplicate in column order but we have to output them in row order.
For symmetry we accumulate both rows rather than only the second row.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 30 Apr 2015 15:23:56 +0000 (16:23 +0100)]
sg-report-job-history: Avoid full runvars table scan (!)
sg-report-job-history wants to know the potential names of runvars
relating to hosts. To do this it tries to find a list of distinct
runvar names which exist in the flights it's processing.
However, it fails to limit the runvar query appropriately, and as a
result postgresql must scan almost the complete runvars table to
produce an answer. This is very slow if the table is bigger than the
database server's RAM.
Fix this by limiting the runvars table query to relevant flights.
Specifically:
* Break the `100' from the LIMIT clause on the flights search
into a local variable $limit.
* Break the bulk of the flights search sql statement text into
a local variable $fromstuff.
* In the runvars statement, add a condition on flights which uses
LIMIT and OFFSET, based on results of the the flights query.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 30 Apr 2015 15:20:51 +0000 (16:20 +0100)]
cs-bisection-step: Use pbm tools, not graphicsmagick/imagemagick
Graphicsmagick / imagemagick have very poor performance with images
with large pixel sizes. The bisector can generate some very large
images.
In an example I have seen, a 21595x21048 png, occupying only 2.6Mby of
disk space. An invocation of `convert' to resize this was using 3Gby
of RAM and lots of CPU. Whereas, the pbm utilities can process this
with much less memory and a tiny fraction of the cpu time.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 29 Apr 2015 15:06:29 +0000 (16:06 +0100)]
ts-kernel-build: Enable CONFIG_SCSI_SAS_ATA
(Some) SAS storage controller drivers do not recognise attached SATA
disks when this option is not set. It is inexplicably not set by
default in Linux 3.14.36 (at least).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 21 Apr 2015 16:39:24 +0000 (17:39 +0100)]
target_cmd_build: Delete build-ok-stamp before starting
Many of the callers of target_cmd_build use a build-ok-stamp idiom to
detect failed builds. This idiom does not work if the stamp file
exists already, so delete it.
In the future we may move more of the test build-ok-stamp, echo ok,
into TestSupport, but this will do for now.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 21 Apr 2015 15:28:37 +0000 (16:28 +0100)]
ts-kernel-build: Enable x86 IOMMU options
This has a variety of beneficial implications:
* The kernel becomes more like the kind of distro kernels that Xen
users are probably using.
* We are more likely to discover any bugs in Linux where Linux
running under Xen (eg as dom0) fights with Xen for control of io
mediation resources or otherwise mishandles the situation.
* A pleasant side effect is that in a kernel which does not yet have
"config: Enable NEED_DMA_MAP_STATE when SWIOTLB is selected"
(a bugfix), enabling INTEL_IOMMU has the side effect of enabling
NEED_DMA_MAP_STATE and thus working around the bug.
The list of options to enable was derived by eyeballing
drivers/iommu/Kconfig from 3.14.34.
I will leave the question of whether to enable any ARM IOMMU options
for the Xen ARM folks to consider.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Boris Ostrovsky <boris.ostrovsky@oracle.com> CC: David Vrabel <david.vrabel@citrix.com> CC: Andrew Cooper <andrew.cooper3@citrix.com> CC: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Tue, 24 Mar 2015 14:23:58 +0000 (14:23 +0000)]
Arrange for core dumps to be placed in /var/core and collect them
Refactor the $kvp_replace helper in ts-xen-install into a generic
helper (which requires using ::EO and ::EI for namespacing) for use
with target_editfile and use it to edit /etc/sysctl.conf to set
kernel.core_pattern on boot.
Tested in standalone mode by installing and running a C program
containing "*(int *)0 = 1;" which, after running "ulimit -c unlimited"
produces the expected core file. ts-logs-capture when run in
standalone mode then picks them up.
I've not yet figured out how to make the desired rlimit take affect
for all processes (including e.g. daemons spawned on boot). Likely
this will involve some combination of pam_limits.so PAM module and
adding explicit ulimit calls to the initscripts which we care about
(primarily xencommons and libvirt initscripts).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 1 Apr 2015 13:24:00 +0000 (14:24 +0100)]
cambridge: Stop publishing logs to chiark
http://osstest.cam.xci-test.com/~osstest/testlogs already exists and
points to the live logs directory, so switch PubBaseUrl to that in the
Cambridge config such that email reports etc contain it. This won't be
externally accessible but I think that won't matter now that the
master production instance is elsewhere.
Arrange that cr-publish-flight-logs doesn't publish the corresponding
thing if either LogsPublish or ResultsPublish is not set, and unset
them in the Cambridge config.
Likewise arrange that cr-ensure-disk-space doesn't do anything if the
configuration variable passed as an option is not set, and unset
Publish (the base for {Logs,Results}Publish) in the Cambridge config.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Check the config variable and not its name.
v3: Adjust for control VM move to xs.citrite.net
Ian Campbell [Wed, 1 Apr 2015 13:12:51 +0000 (14:12 +0100)]
cambridge: Do not try to push harness to XenProject instance output
By arranging for cr-publish-flight-logs to ignore --push-harness if
either of HarnessPublishGitRepoDir or HarnessPublishGitUserHost are
not specified
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2:
- Avoid logm which isn't available here, wasn't saying much of use
anyway.
- Syntax fix (is not a function, so exit not return)
Perhaps we should have our own tree for such things, but for now just
nobble it.
Ian Campbell [Wed, 1 Apr 2015 12:50:50 +0000 (13:50 +0100)]
Handle osstest's own local push gate in non-master production instances
We want to arrange that the master XenProject instance continues to
test its own pretest branch while any downstream instances will pickup
changes from the master instance's production (i.e. tested) branch,
which is published at git://xenbits.xen.org/osstest.git#master. We
want to also be able to use local pretest for local changes (which may
or may not get merged back upstream).
Add a new configuration option OsstestUpstream which by default is
"git://xenbits.xen.org/osstest.git master" and which is cleared to
nothing on the master instance via production-config.
If the option is not set then the existing behaviour is unchanged.
If the option is set then osstest branch flights will still prefer to
test the local pretest branch, but if nothing is pending there then it
will proceed by merging the upstream branch into the local production
branch and testing the result.
This merge must be done:
- in a clone not in the main testing.git in order to avoid inserting
merge conflict markers into the active set of scripts.
- in a non-bare repo because git merge requires it.
$repos/osstest is a bare repo which we want to keep that way because
using repo_tree_rev_fetch_git to fetch the remote branch is
convenient.
So we use $repos/osstest-merge as a temporary merge repo and reclone
from the active local repo each time.
All of this happens in ap-fetch-version.
As part of this arrange that the result is always left in the ap-fetch
branch of the for-osstest.git repo (even for existing cases) and the
sha1 is produced as output. Resetting to that revision is handled by
cr-daily-branch.
If the merge fails then manual intervention (i.e. a manual merge and
push to the _local_ pretest) will be required. Likewise if local
pretest and local production have diverged manual intervention will be
required.
In ap-push we stop pushing to xenbits#master except for the master
instance if an upstream is defined. At some point it might be useful
to add a configuration option for where to push to but I don't have
that requirement right now.
ap-fetch-version-old requires no changes.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
v4:
- Use git update-ref properly, i.e. with the full ref name, otherwise
it creates random .git/ap-fetch refs
v3:
- Only merge from upstream if there is nothing pending locally.
- Always update ap-fetch.
v2:
- Arrange for $OSSTEST_USE_HEAD=y to take precendence
- drop LOCALREV (which was wrong anyway) in favour of inline
branchname
- Rename OSSTEST_REVISION_MERGE as revision_merge to avoid implying
it can be set and will be honoured.
- Git in Debian Squeeze lacks -C and --no-edit, adjust accordingly.
Ian Campbell [Fri, 1 May 2015 10:20:45 +0000 (11:20 +0100)]
Osstest/Debian.pm: Use Fqdn hostprop when collecting host keys
Otherwise hosts which are not in the same DnsDomain are not processed,
resulting in log messages such as:
2015-05-01 10:06:19 Z skipping host key for nonexistent host marilith-n4.xs.citrite.net
2015-05-01 10:06:20 Z skipping host key for nonexistent host lace-bug.xs.citrite.net
The practical impact of this appears to be that the pair migration
tests can fail with:
Ian Campbell [Wed, 29 Apr 2015 10:14:58 +0000 (11:14 +0100)]
cambridge: Switch configuration to use osstest.xs.citrite.net
The VM has moved to different infrastructure and its new name is
osstest.citrite.net.
Update ExecutiveDbnamePat. The DB is still in the XC infrastructure so
using DnsDomain (the default) no longer works.
Set {Owner,Queue}DaemonHost to refer to the new VM host and not the
default ControlDaemonHost value of control-daemons.osstest.cam.xci-test.com
(which will be removed later).
We set both variables rather than just ControlDaemonHost in case we
ever want to move one but not the other.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Wed, 1 Apr 2015 10:54:47 +0000 (11:54 +0100)]
allow instance specific settings
cri-args-hostlists and invoke-daemon now check for
$HOME/.xen-osstest/settings which can contain things like "export
OSSTEST_CONFIG=production-config-cambridge" to tailor things for a
particular instance of osstest running in production mode.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
[ ijc -- add invoke-daemon too and reword commit message accordingly ]
FreeBSD: Cleanups relating to guest images and ts-freebsd-install script
Remove some unused variables from ts-freebsd-install script. Also make the
third parameter of target_put_guest_image optional and fix both callers of
this function.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
New 10.1 images are larger than the previous 10.0 images, so change
the size of the LVM volume to accommodate them, in preparation.
Increase the size to 24000 in case of future increases upstream.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
projects / people / liuw / osstest.git / log