Ian Campbell [Tue, 31 Mar 2015 15:06:46 +0000 (16:06 +0100)]
tcl: Handle environment variables which are unset.
This allows wrappers such as the standalone wrapper to do
OSSTEST_SIMULATE=$foo ./sg-run-job
and not worry if $foo is unset.
Do likewise for OSSTEST_TCL_JOBDB_DEBUG.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 13 Mar 2015 15:19:43 +0000 (15:19 +0000)]
HostnameSortSwapWords: Make name order mangling configurable
We still default to having the mangling enabled. Arguably this is
wrong I'm am minimising the number of things that will be wrong for
the existing Cambridge instance.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 9 Mar 2015 16:22:58 +0000 (12:22 -0400)]
cr-ensure-disk-space: Permit argument to specify local directory.
If the argument is Logs rather than LogsPublish (ie, refers to a local
directory (without `:') rather than a remote one (with `:'), do things
locally (by invoking sh -ec so that we have identical quoting rules to
ssh).
No effectively functional change with the current configuration.
We still always do a local deletion. This is anomalous and will
disappear shortly.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Sun, 8 Mar 2015 12:20:34 +0000 (12:20 +0000)]
daemons: Allow QueueDaemon and OwnerDaemon to run on different hosts
We want the OwnerDaemon to run on the same host as the database (for
fate-sharing reasons). OTOH the QueueDaemon is less critical if it
fails, and it generates reports etc., and wants to be more frequently
updated, so it should run on the osstest VM.
Permit this by:
* Providing OwnerDaemonHost and QueueDaemonHost config settings
which default to the value of ControlDaemonHost.
* Using those everywhere.
* In the daemons' Tcl code, have main-daemon take the string `Owner'
or `Queue' so that it can look up both the host and port.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 6 Mar 2015 18:39:44 +0000 (18:39 +0000)]
pxe setup: New TftpPxeTemplatesReal feature
Rather than having mg-hosts mkpxedir hardcode the strange thing done
in the XenClient test lab in the Citrix Cambridge office, provide a
somewhat more general and correct approach:
* Generalise host_pxefile to support [Tftp]PxeTemplatesReal as well
as [Tftp]PxeTemplates.
* Default [Tftp]PxeTemplatesReal to ''
mg-hosts mkpxedir now uses these templates, as follow:
* Create the host's PxeTemplates-based pxe file's parent
directories and make the parent directory be owned by PxeGroup.
* If the PxeTemplatesReal is specified and different, make a symlink
named according to PxeTemplatesReal pointing at the previous file.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 5 Mar 2015 19:04:43 +0000 (14:04 -0500)]
PDU: pdu-msw: Support APC v6 firmware
APC PDUs with firmware 6.x have a different OID space for turning
ports on and off, to the one for querying. The old namespace still
works to turn the port on and off but returns a genErr error response!
Support a new command-line option --apc6 to use this other OID.
Ian Jackson [Thu, 5 Mar 2015 19:02:09 +0000 (14:02 -0500)]
PDU: pdu-msw: Split $read_oid and $write_oid
Some PDUs have a different OID space for turning ports on and off, to
the one for querying. To make this easier to handle, split the
variable $oid into $read_oid and $write_oid.
Also move $baseoid settings earlier so that we can modify them with
command-line arguments.
No functional change in this patch.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Wei Liu [Mon, 2 Feb 2015 19:57:13 +0000 (19:57 +0000)]
mfi-common, make-flight: create XSM test jobs
Duplicate Debian PV and HVM test jobs for XSM testing.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v8:
1. Make libvirtbuildjob = ${bfi}build-$dom0arch-libvirt
Changes in v6:
1. Skip generating xsm job for different platforms.
2. Use "xsms".
3. Reformat some long lines.
Wei Liu [Mon, 2 Feb 2015 19:53:26 +0000 (19:53 +0000)]
make-flight: factor out do_pv_debian_tests
Pure code motion. No effect on job generation.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Wei Liu [Fri, 12 Sep 2014 15:29:00 +0000 (16:29 +0100)]
Debian.pm: load flask policy in uboot
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v15:
1. Use new flask option "flask=enforcing".
Changes in v10:
1. Correctly get $flaskpolicy.
Changes in v9:
1. Add "xen,multiboot-module".
Changes in v8:
1. Append flask_enforcing=1 and flask_enabled=1.
Wei Liu [Mon, 23 Feb 2015 12:03:16 +0000 (12:03 +0000)]
ts-xen-build: only move hypervisor to xeninstall
... so that we can leave xenpolicy-* in tools tarball.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v13:
1. Use find rune to get list of files to move.
Wei Liu [Mon, 8 Sep 2014 15:06:52 +0000 (16:06 +0100)]
ts-xen-build: build with XSM support if requested
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes in v14:
1. Use target_cmd_output instead of target_cmd_output_root.
Changes in v5:
1. Only set XSM_ENABLE when runvar is defined.
2. Fix inconsistent whitespace.
Wei Liu [Sun, 12 Oct 2014 16:04:34 +0000 (17:04 +0100)]
overlay: update overlay/etc/grub.d/20_linux_xen
This file was originally created to work around Debian bug #633127
("/etc/grub/20_linux does not recognise some old Xen kernels").
According to Debian bug tracker [0], #633127 bug is fixed in Wheezy. As
we're now using Wheezy in OSSTest we can safely remove the old overlay
file if there's no further bugs discovered.
However we have another bug #690538 ("grub-common: Please make submenu
creation optional or at least allow users to disable it easily") that
would break OSSTest. We're now using Wheezy in production. There's no
way to disable submenu in Wheezy. And submenu breaks OSSTest's grub menu
parser.
So update this overlay file to the one in Wheezy's grub-common
1.99-27+deb7u2 and take care of Debian bug #690538 by removing the lines
to generate submenu.
Also work around GRUB bug #43420 ("20_linux_xen doesn't support Xen XSM
policy file") by applying a small patch proposed in [2].
Add a note to reference #633127 and #690538 above grub2 setup function.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Cc: Ian Jackson <ian.jackson@eu.citrix.com> Cc: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
Changes in v15:
1. Use new flask option "flask=enforcing".
Ian Campbell [Wed, 18 Feb 2015 10:43:26 +0000 (10:43 +0000)]
Debian: Add "fdt chosen" to boot script
This causes u-boot to fill in the various fields in the chosen node
(specifically the bootargs) which would otherwise not be done until
the bootz command. Doing it manually means the following "fdt print
/chosen" will print what is actually going to be used.
This change means that instead of whatever /chosen/bootargs is
embedded in the firmware FDT we end up printing what we will actually
use.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Fri, 6 Mar 2015 10:07:49 +0000 (10:07 +0000)]
allow: Allow rumpuserxen-demo-xenstorels/xenstorels failures on all branches
This test is subject to a heisenbug. The next step after this one is a
repetition (50 times) of the same test which should reliably hit the
issue.
In the meantime this individual attempt sometimes hits and sometimes
doesn't, causing unhelpful false regressions.
The repeated test has never passed in any branch and so is not yet
considered a regression.
Since the repeated test will catch the actual heisenbug I haven't
bothered to add a rumpuserxen specific override which makes this step
blocking again.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 3 Mar 2015 13:13:02 +0000 (13:13 +0000)]
allow files: Support negation syntax
We now support !<pattern>, which undoes the effect of a <pattern>
specified previously (perhaps in another file).
Note that the logic is a bit poor: !... only undoes the effect of an
identical (or equivalent) earlier pattern; it doesn't provide a
specific override.
Improving that would involve a more invasive change to
sg-report-flight which I don't want to do right now.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 17 Feb 2015 17:35:57 +0000 (17:35 +0000)]
NTP servers: Work around Debian's failure to honour preseed
Setting clock-setup/ntp-server is not sufficient: it only takes effect
in the installer (!)
I have reported this as Debian #778564. In the meantime we should
work around it for current releases (including jessie, which is
frozen).
For later releases, the new ntp.conf editing code arranges to bomb out
if we have an NTP server configured and find it hasn't been honoured
during the install.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
-### END OF DEBIAN PRESEED BASE
+END
+
+ my $ntpserver = get_target_property($ho,'NtpServer');
+use Data::Dumper;
+print STDERR "PRESEED NTP ", Dumper($ntpserver);
+ $preseed .= <<"END" if $ntpserver;
+d-i clock-setup/ntp-server string $ntpserver
+END
+ $preseed .= <<"END";
+
+### END OF DEBIAN PRESEED BASE
END
-}
+
+ return $preseed;
+}
sub preseed_create ($$;@) {
my ($ho, $sfx, %xopts) = @_;
diff --git a/README b/README
index 6e63e97..0a0242c 100644
--- a/README
+++ b/README
@@ -334,6 +334,11 @@ HostProp_<testbox>_TftpScope
Defines the Tftp scope (i.e. subnet) where this host resides. See
"TftpFoo_<scope> and TftpFoo" below.
+HostProp_<testbox>_NtpServer
+ NTP server to use. You should probably have your own local
+ NTP server for production use; the default is to use the operating
+ system's default (normally, Debian's pool.ntp.org servers).
+
HostFlags_<testbox>
Defines a set of flags for the host. Flags is a list separated by
whitespace, comma or semi-colon. A flag can be unset by prepending
@@ -357,9 +362,7 @@ HostGroupFlags_<group>
merged with the host specific flags. Only used in standalone mode.
DebianPreseed
- Text to add to the debian-installer preseed file. Optional
- but you will need to set some NTP servers here if your firewall
- doesn't permit NTP to Debian's pool.ntp.org servers.
+ Text to add to the debian-installer preseed file. Optional.
-### END OF DEBIAN PRESEED BASE
+END
+
+ my $ntpserver = get_target_property($ho,'NtpServer');
+use Data::Dumper;
+print STDERR "PRESEED NTP ", Dumper($ntpserver);
+ $preseed .= <<"END" if $ntpserver;
+d-i clock-setup/ntp-server string $ntpserver
+END
+ $preseed .= <<"END";
+
+### END OF DEBIAN PRESEED BASE
END
-}
+
+ return $preseed;
+}
sub preseed_create ($$;@) {
my ($ho, $sfx, %xopts) = @_;
diff --git a/README b/README
index 6e63e97..0a0242c 100644
--- a/README
+++ b/README
@@ -334,6 +334,11 @@ HostProp_<testbox>_TftpScope
Defines the Tftp scope (i.e. subnet) where this host resides. See
"TftpFoo_<scope> and TftpFoo" below.
+HostProp_<testbox>_NtpServer
+ NTP server to use. You should probably have your own local
+ NTP server for production use; the default is to use the operating
+ system's default (normally, Debian's pool.ntp.org servers).
+
HostFlags_<testbox>
Defines a set of flags for the host. Flags is a list separated by
whitespace, comma or semi-colon. A flag can be unset by prepending
@@ -357,9 +362,7 @@ HostGroupFlags_<group>
merged with the host specific flags. Only used in standalone mode.
DebianPreseed
- Text to add to the debian-installer preseed file. Optional
- but you will need to set some NTP servers here if your firewall
- doesn't permit NTP to Debian's pool.ntp.org servers.
+ Text to add to the debian-installer preseed file. Optional.
Ian Jackson [Tue, 17 Feb 2015 17:21:47 +0000 (17:21 +0000)]
TestSupport: Provide get_target_property
This looks in the supplied $ho, but if that's a $gho (ie it has a
$gho->{Host}) it also looks in its host.
This is going to be useful for a fair variety of host-specific or
infrastructure-determined properties.
It seems to me that whether a property ought to be looked up in the
host if not found in a particular guest depends mostly on the
property, and not on the way the guest is configured. The easiest way
to represent that in the osstest codebase is probably to call
get_target_property instead of get_host_property in the appropriate
places.
The use of recursion will make this look through a series of nested
hosts if we have nested virtualisation going on. Indeed, nested
virtualisation may benefit from replacement of get_host_property by
get_target_property in a number of cases. (At the time of writing
there is no nested virt in osstest mainline, but it's on the way.)
Currently there are no callers of get_target_property. One will
appear shortly.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Robert Hu <robert.hu@intel.com> CC: LongtaoX Pang <longtaox.pang@intel.com>
Ian Jackson [Fri, 6 Feb 2015 17:09:40 +0000 (17:09 +0000)]
rump kernel tests: Repeat the xenstorels test 50 times
Add a new step which uses repeat-ts to run
ts-rumpuserxen-demo-xenstorels many times.
We have to run ts-guest-destroy-hard after each time, to destroy the
guest which the demo script leaves lying about.
Strategically placed `+'s in the repeat-ts command line arrange that
the testid ends up being
rumpuserxen-demo-xenstorels/xenstorels.repeat
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Run the test after, rather than before, the explicit
ts-guest-destroy-hard. That will avoid blocking the single
destroy test if the repeat fails.
No longer specify to tolerate failures of the post-run-demo
destroy, as if the test passes so must the destroy. Now by-hand
testing may need a different ts-repeat-test rune, but in practice
by-hand testing will probably involve a shell loop or something
anyway.
Ian Jackson [Fri, 6 Feb 2015 17:08:31 +0000 (17:08 +0000)]
rump kernel tests: Cancel unneeded edits of guest config
If the guest config is already set up to preserve, cancel the edit.
We are going to repeat this test, and this avoids creating many
identical copies of the same file in the log output.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbelL@citrix.com>
Ian Jackson [Fri, 6 Feb 2015 15:36:34 +0000 (15:36 +0000)]
sg-run-job: testid generation: Process ts more like rest of args
Remove ts as a separate parameter to spawn-ts. The test script now
becomes the first entry in args.
We process it through the arg loop as before. Currently there are no
calls where the first arg is `+' so the test script name ends up in
both real_args and testid_args.
We split it out of real_args into the ts variable with lshift.
We split it out of testid_args into the deftestid with lshift.
So afterwards in spawn-ts, all the variables (including real_args, ts,
deftestid and testid_args and hence host_testid_suffix) have the
values they would have had before.
Therefore there is no functional change for any existing calls.
However, because the first argument is not treated specially for the
`+' procesing loop, it is now possible to specify `+' as the first
entry in args to spawn-ts (ie where ts used to be) to arrange that the
deftestid (and hence, probably, the testid) is computed using later
arguments.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Mon, 9 Feb 2015 17:55:05 +0000 (17:55 +0000)]
sg-report-job-history: Show which host test ran on
Add a column listing the host(s) used. We first find the relevant set
of host runvars (with a SELECT DISTINCT) and then look up each var for
each actual flight.
We do the pattern-matching on runvar names in perl to avoid giving the
postgresql optimiser a chance to turn this query into a full table
scan of the runvars table. (A previous iteration of this patch
searched, in the per-flight loop, for all runvars whose name was
appropriately LIKE, with appalling performance.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 9 Feb 2015 14:22:50 +0000 (14:22 +0000)]
allow.all: Do not regard libvirt guest start failures as regressions
We have been experiencing some regressions in test-amd64-*-libvirt
guest-start.
These appear to be host-specific: for example, test-amd64-i386-libvirt
succeeded in 34278 on itch-mite but failed in 34234 on bush-cricket.
The only difference in osstest's records of the versions is a change
to linuxfirmware.
I booked out field-cricket and have repro'd some problems there. It
appears that libvirtd can crash. The symptoms are not the same as in
the push gate failures in osstest (34316, 34234) and linux-linus
(34299) but similar to those in linux-3.0 (34291) and linux-3.16
(34285).
When running libvirtd in the foreground we can get this:
libvirtd: libxl_event.c:1791: libxl__ao_complete_check_progress_reports: Assertion `ao->in_initiator' failed.
So, it appears to be some kind of race. The libvirt tests are new so
this is not a real regression in any of the trees gated by osstest.
So, permit this test for now.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Campbell [Fri, 30 Jan 2015 10:55:03 +0000 (10:55 +0000)]
README.dev: Updates to commissioning process
- mkpxedir requires the resource to be allocated
- mention the need to bless the hosts for comissioning run, and to add
proper blessings afterwards.
- run mg-execute-flight in screen
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 4 Feb 2015 09:49:44 +0000 (09:49 +0000)]
ts-logs-capture: ensure captured log files are readable.
At least /var/log/libvirt/libvirtd.log is 0600 on the target machine
and this gets preserved in the captured logfile, eventually meaning
that apache on the log server cannot read the file to serve it ("You
don't have permission to access...").
Set all logfiles to 0666, modified by umask.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
projects / osstest.git / log