Ian Jackson [Wed, 6 May 2015 22:47:30 +0000 (23:47 +0100)]
cs-bisection-step: Abandon repro attempts after a bit
If we have had a number of attempts at a repro, and none of them have
produced a pass or fail, something is probably wrong and we should
give up rather than carrying on.
Handle this with the machinery we use for conflicting test results.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 6 May 2015 22:41:00 +0000 (23:41 +0100)]
cs-bisection-step: Report conflict if basis pass/fail are wrong
It can happen that the (for example) supposed basis pass (originally
only tested on another host) failed (when reproduced, or for some
other reason). When that happens do not attempt to get it to pass;
instead, treat it the same way we would if we had actually got
conflicting results at that revision.
(Conversely, do not attempt to get a basis fail if the basis fail has
already passed on the selected host. This is, as it happens,
impossible in a bisection triggered by sg-report-flight with the
current invocation arrangements - but cs-bisection-step should
handle it correctly.)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 1 May 2015 14:54:03 +0000 (15:54 +0100)]
sg-report-flight: Report stepno and testid of first worst fail
This makes reading the scoreboard considerably easier.
We abuse the local variable @worst slightly, pushing the extra info we
are going to print onto the end of it.
We also have to defer printing the cells, because we compute the cell
to duplicate in column order but we have to output them in row order.
For symmetry we accumulate both rows rather than only the second row.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 30 Apr 2015 15:23:56 +0000 (16:23 +0100)]
sg-report-job-history: Avoid full runvars table scan (!)
sg-report-job-history wants to know the potential names of runvars
relating to hosts. To do this it tries to find a list of distinct
runvar names which exist in the flights it's processing.
However, it fails to limit the runvar query appropriately, and as a
result postgresql must scan almost the complete runvars table to
produce an answer. This is very slow if the table is bigger than the
database server's RAM.
Fix this by limiting the runvars table query to relevant flights.
Specifically:
* Break the `100' from the LIMIT clause on the flights search
into a local variable $limit.
* Break the bulk of the flights search sql statement text into
a local variable $fromstuff.
* In the runvars statement, add a condition on flights which uses
LIMIT and OFFSET, based on results of the the flights query.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 30 Apr 2015 15:20:51 +0000 (16:20 +0100)]
cs-bisection-step: Use pbm tools, not graphicsmagick/imagemagick
Graphicsmagick / imagemagick have very poor performance with images
with large pixel sizes. The bisector can generate some very large
images.
In an example I have seen, a 21595x21048 png, occupying only 2.6Mby of
disk space. An invocation of `convert' to resize this was using 3Gby
of RAM and lots of CPU. Whereas, the pbm utilities can process this
with much less memory and a tiny fraction of the cpu time.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 29 Apr 2015 15:06:29 +0000 (16:06 +0100)]
ts-kernel-build: Enable CONFIG_SCSI_SAS_ATA
(Some) SAS storage controller drivers do not recognise attached SATA
disks when this option is not set. It is inexplicably not set by
default in Linux 3.14.36 (at least).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 21 Apr 2015 16:39:24 +0000 (17:39 +0100)]
target_cmd_build: Delete build-ok-stamp before starting
Many of the callers of target_cmd_build use a build-ok-stamp idiom to
detect failed builds. This idiom does not work if the stamp file
exists already, so delete it.
In the future we may move more of the test build-ok-stamp, echo ok,
into TestSupport, but this will do for now.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 21 Apr 2015 15:28:37 +0000 (16:28 +0100)]
ts-kernel-build: Enable x86 IOMMU options
This has a variety of beneficial implications:
* The kernel becomes more like the kind of distro kernels that Xen
users are probably using.
* We are more likely to discover any bugs in Linux where Linux
running under Xen (eg as dom0) fights with Xen for control of io
mediation resources or otherwise mishandles the situation.
* A pleasant side effect is that in a kernel which does not yet have
"config: Enable NEED_DMA_MAP_STATE when SWIOTLB is selected"
(a bugfix), enabling INTEL_IOMMU has the side effect of enabling
NEED_DMA_MAP_STATE and thus working around the bug.
The list of options to enable was derived by eyeballing
drivers/iommu/Kconfig from 3.14.34.
I will leave the question of whether to enable any ARM IOMMU options
for the Xen ARM folks to consider.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Boris Ostrovsky <boris.ostrovsky@oracle.com> CC: David Vrabel <david.vrabel@citrix.com> CC: Andrew Cooper <andrew.cooper3@citrix.com> CC: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Tue, 24 Mar 2015 14:23:58 +0000 (14:23 +0000)]
Arrange for core dumps to be placed in /var/core and collect them
Refactor the $kvp_replace helper in ts-xen-install into a generic
helper (which requires using ::EO and ::EI for namespacing) for use
with target_editfile and use it to edit /etc/sysctl.conf to set
kernel.core_pattern on boot.
Tested in standalone mode by installing and running a C program
containing "*(int *)0 = 1;" which, after running "ulimit -c unlimited"
produces the expected core file. ts-logs-capture when run in
standalone mode then picks them up.
I've not yet figured out how to make the desired rlimit take affect
for all processes (including e.g. daemons spawned on boot). Likely
this will involve some combination of pam_limits.so PAM module and
adding explicit ulimit calls to the initscripts which we care about
(primarily xencommons and libvirt initscripts).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
FreeBSD: Cleanups relating to guest images and ts-freebsd-install script
Remove some unused variables from ts-freebsd-install script. Also make the
third parameter of target_put_guest_image optional and fix both callers of
this function.
New 10.1 images are larger than the previous 10.0 images, so change
the size of the LVM volume to accommodate them, in preparation.
Increase the size to 24000 in case of future increases upstream.
Ian Campbell [Tue, 31 Mar 2015 15:06:46 +0000 (16:06 +0100)]
tcl: Handle environment variables which are unset.
This allows wrappers such as the standalone wrapper to do
OSSTEST_SIMULATE=$foo ./sg-run-job
and not worry if $foo is unset.
Do likewise for OSSTEST_TCL_JOBDB_DEBUG.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 13 Mar 2015 15:19:43 +0000 (15:19 +0000)]
HostnameSortSwapWords: Make name order mangling configurable
We still default to having the mangling enabled. Arguably this is
wrong I'm am minimising the number of things that will be wrong for
the existing Cambridge instance.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 9 Mar 2015 16:22:58 +0000 (12:22 -0400)]
cr-ensure-disk-space: Permit argument to specify local directory.
If the argument is Logs rather than LogsPublish (ie, refers to a local
directory (without `:') rather than a remote one (with `:'), do things
locally (by invoking sh -ec so that we have identical quoting rules to
ssh).
No effectively functional change with the current configuration.
We still always do a local deletion. This is anomalous and will
disappear shortly.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Sun, 8 Mar 2015 12:20:34 +0000 (12:20 +0000)]
daemons: Allow QueueDaemon and OwnerDaemon to run on different hosts
We want the OwnerDaemon to run on the same host as the database (for
fate-sharing reasons). OTOH the QueueDaemon is less critical if it
fails, and it generates reports etc., and wants to be more frequently
updated, so it should run on the osstest VM.
Permit this by:
* Providing OwnerDaemonHost and QueueDaemonHost config settings
which default to the value of ControlDaemonHost.
* Using those everywhere.
* In the daemons' Tcl code, have main-daemon take the string `Owner'
or `Queue' so that it can look up both the host and port.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 6 Mar 2015 18:39:44 +0000 (18:39 +0000)]
pxe setup: New TftpPxeTemplatesReal feature
Rather than having mg-hosts mkpxedir hardcode the strange thing done
in the XenClient test lab in the Citrix Cambridge office, provide a
somewhat more general and correct approach:
* Generalise host_pxefile to support [Tftp]PxeTemplatesReal as well
as [Tftp]PxeTemplates.
* Default [Tftp]PxeTemplatesReal to ''
mg-hosts mkpxedir now uses these templates, as follow:
* Create the host's PxeTemplates-based pxe file's parent
directories and make the parent directory be owned by PxeGroup.
* If the PxeTemplatesReal is specified and different, make a symlink
named according to PxeTemplatesReal pointing at the previous file.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 5 Mar 2015 19:04:43 +0000 (14:04 -0500)]
PDU: pdu-msw: Support APC v6 firmware
APC PDUs with firmware 6.x have a different OID space for turning
ports on and off, to the one for querying. The old namespace still
works to turn the port on and off but returns a genErr error response!
Support a new command-line option --apc6 to use this other OID.
Ian Jackson [Thu, 5 Mar 2015 19:02:09 +0000 (14:02 -0500)]
PDU: pdu-msw: Split $read_oid and $write_oid
Some PDUs have a different OID space for turning ports on and off, to
the one for querying. To make this easier to handle, split the
variable $oid into $read_oid and $write_oid.
Also move $baseoid settings earlier so that we can modify them with
command-line arguments.
No functional change in this patch.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Wei Liu [Mon, 2 Feb 2015 19:57:13 +0000 (19:57 +0000)]
mfi-common, make-flight: create XSM test jobs
Duplicate Debian PV and HVM test jobs for XSM testing.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v8:
1. Make libvirtbuildjob = ${bfi}build-$dom0arch-libvirt
Changes in v6:
1. Skip generating xsm job for different platforms.
2. Use "xsms".
3. Reformat some long lines.
Wei Liu [Mon, 2 Feb 2015 19:53:26 +0000 (19:53 +0000)]
make-flight: factor out do_pv_debian_tests
Pure code motion. No effect on job generation.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Wei Liu [Fri, 12 Sep 2014 15:29:00 +0000 (16:29 +0100)]
Debian.pm: load flask policy in uboot
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v15:
1. Use new flask option "flask=enforcing".
Changes in v10:
1. Correctly get $flaskpolicy.
Changes in v9:
1. Add "xen,multiboot-module".
Changes in v8:
1. Append flask_enforcing=1 and flask_enabled=1.
Wei Liu [Mon, 23 Feb 2015 12:03:16 +0000 (12:03 +0000)]
ts-xen-build: only move hypervisor to xeninstall
... so that we can leave xenpolicy-* in tools tarball.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v13:
1. Use find rune to get list of files to move.
Wei Liu [Mon, 8 Sep 2014 15:06:52 +0000 (16:06 +0100)]
ts-xen-build: build with XSM support if requested
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v14:
1. Use target_cmd_output instead of target_cmd_output_root.
Changes in v5:
1. Only set XSM_ENABLE when runvar is defined.
2. Fix inconsistent whitespace.
Wei Liu [Sun, 12 Oct 2014 16:04:34 +0000 (17:04 +0100)]
overlay: update overlay/etc/grub.d/20_linux_xen
This file was originally created to work around Debian bug #633127
("/etc/grub/20_linux does not recognise some old Xen kernels").
According to Debian bug tracker [0], #633127 bug is fixed in Wheezy. As
we're now using Wheezy in OSSTest we can safely remove the old overlay
file if there's no further bugs discovered.
However we have another bug #690538 ("grub-common: Please make submenu
creation optional or at least allow users to disable it easily") that
would break OSSTest. We're now using Wheezy in production. There's no
way to disable submenu in Wheezy. And submenu breaks OSSTest's grub menu
parser.
So update this overlay file to the one in Wheezy's grub-common
1.99-27+deb7u2 and take care of Debian bug #690538 by removing the lines
to generate submenu.
Also work around GRUB bug #43420 ("20_linux_xen doesn't support Xen XSM
policy file") by applying a small patch proposed in [2].
Add a note to reference #633127 and #690538 above grub2 setup function.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
Changes in v15:
1. Use new flask option "flask=enforcing".
Ian Campbell [Wed, 18 Feb 2015 10:43:26 +0000 (10:43 +0000)]
Debian: Add "fdt chosen" to boot script
This causes u-boot to fill in the various fields in the chosen node
(specifically the bootargs) which would otherwise not be done until
the bootz command. Doing it manually means the following "fdt print
/chosen" will print what is actually going to be used.
This change means that instead of whatever /chosen/bootargs is
embedded in the firmware FDT we end up printing what we will actually
use.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Fri, 6 Mar 2015 10:07:49 +0000 (10:07 +0000)]
allow: Allow rumpuserxen-demo-xenstorels/xenstorels failures on all branches
This test is subject to a heisenbug. The next step after this one is a
repetition (50 times) of the same test which should reliably hit the
issue.
In the meantime this individual attempt sometimes hits and sometimes
doesn't, causing unhelpful false regressions.
The repeated test has never passed in any branch and so is not yet
considered a regression.
Since the repeated test will catch the actual heisenbug I haven't
bothered to add a rumpuserxen specific override which makes this step
blocking again.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 3 Mar 2015 13:13:02 +0000 (13:13 +0000)]
allow files: Support negation syntax
We now support !<pattern>, which undoes the effect of a <pattern>
specified previously (perhaps in another file).
Note that the logic is a bit poor: !... only undoes the effect of an
identical (or equivalent) earlier pattern; it doesn't provide a
specific override.
Improving that would involve a more invasive change to
sg-report-flight which I don't want to do right now.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 17 Feb 2015 17:35:57 +0000 (17:35 +0000)]
NTP servers: Work around Debian's failure to honour preseed
Setting clock-setup/ntp-server is not sufficient: it only takes effect
in the installer (!)
I have reported this as Debian #778564. In the meantime we should
work around it for current releases (including jessie, which is
frozen).
For later releases, the new ntp.conf editing code arranges to bomb out
if we have an NTP server configured and find it hasn't been honoured
during the install.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>