Ian Campbell [Thu, 1 Oct 2015 15:01:26 +0000 (16:01 +0100)]
Debian: Use dtbs from kernel dist when booting that kernel
The kernel dist built by ts-kernel-build puts the corresponding dtbs
into /boot/dtbs/$kvers.
The host installer's dtbs remain in /boot/dtbs and are used when
booting the native kernel.
It's possible that this change will expose bugs which exist in the
DTBs in previous kernel branches (3.18 and 4.1). I've not
investigated, I think we should accept this possibility and deal with
it via backport requests (and maybe some force pushes if appropriate)
as necessary.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Thu, 1 Oct 2015 15:04:22 +0000 (16:04 +0100)]
Debian: Enable interpolation in uboot_scr_load_dtb here doc
By switching <<'END' to <<END. A future patch is going to want to put
a variable here which requires interpretation by the Perl.
Unfortunately this means lots of extra backslashes to escape things
such that they pass through Perl and Shell and end up as ${foo} in the
resulting u-boot script.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Thu, 1 Oct 2015 10:08:49 +0000 (11:08 +0100)]
ts-kernel-build: Include dtbs in dist file
These are installed to $(INSTALL_PATH)/dtbs/$(KERNEL_RELEASE) where
$(INSTALL_PATH) defaults to /boot but we override it to our staging
/boot.
Note that ts-host-install will install the OS dtbs directly into
/boot/dtbs without the subdirectory, so this won't clash and could be
considered a fallback hence I don't propose to move those ones.
The install_dtbs target has been available since v3.14, wherease we
only test v3.16 onwards on ARM, hence no arrangements are needed to
conditionalise this installation over and above the per-arch
arrangements made here.
Having now set $(INSTALL_PATH) I think the "install" target could now
take over the installation of System.map, vmlinux and .config into
/boot but I've not checked this with all historical kernel versions
and don't intend to make this change now.
Remove any previous dist dir on install, otherwise the kernel tends to
create dist/boot/dtbs.old with the previous contents on repeated use.
Seems like good hygiene anyway.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Sat, 3 Oct 2015 10:47:28 +0000 (11:47 +0100)]
ts-freebsd-install: Pass -s option to kpartx
This is "Sync mode. Don't return until the partitions are created",
which seems to be needed in Jessie. The option also exists in Wheezy,
according to the manpage.
Without this the following mount fails having apparently raced against
the creation of the device nodes.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Cope with this here, by treating the x86/xen (PV) case as the special
case that it is (due to the requirement of a separate i686-pae
installer kernel).
This goes unnoticed in the distros-debian flights because the kernel
is downloaded at runtime via a runvar. It matters once regular
(non-distro-debian) flights run with Jessie because then the d-i which
is used by the di based tests is taken from the result of
mg-debian-installer-update (e.g. for the -qcow2 flights etc).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 22 Sep 2015 09:59:40 +0000 (10:59 +0100)]
ts-freebsd-install: Use $gho->{Lvdev} instead of target_guest_lv_name
prepareguest has already assigned this so we should use it instead of
replicating (perhaps wrongly since target_guest_lv_name and
target_choose_vg can behave differently if multiple vgs are present).
$gho->{Lvdev} has been adjusted to return /dev/mapper/vg-lv paths
which is required to be able to add partition numbers since kpartx
does not create the /dev/vg/lv form.
Ian Campbell [Fri, 2 Oct 2015 09:31:13 +0000 (10:31 +0100)]
TestSupport: Use lv_dev_mapper in guest_find_lv
This has the effect of switching $gho->{Lvdev} from /dev/$VG/$LV to
/dev/mapper/$VG-$LV (where $VG and $LV have an s/-/--/ transformation
applied).
The two paths point to the samedevice and most call sites don't care
about the distinction. Some places which use "kpartx $dev" to Lvdev
expect to be able to append a partition number to $dev, while kpartx
only creates the /dev/mapper form, meaning such places cannot use
Lvdev. By making this switch we allow these places (such as
ts-freebsd-install) to use kpartx.
The only other place I'm aware of which requires one form or the other
is the Debian initramfs code which expects root=/dev/mapper/vg-lv and
does not accept /dev/vg/lv. However this is already handled correctly.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 22 Sep 2015 09:33:00 +0000 (10:33 +0100)]
Debian: uboot: Use di_vg_name() and lv_dev_mapper() for root=
root is not a "guest lv", so using target_guest_lv_name is misleading.
target_guest_lv_name also fails to properly handle the d-i vg name
correctly, which di_vg_name does. Specifically under Jessie an extra
"-vg" is appended to the hostname compared with Wheezy and Squeeze.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 14 Jul 2015 14:32:54 +0000 (15:32 +0100)]
ms-flights-summary: Produce an HTML report of all active flights
Jobs are categorised by a new ->Job field. This is added by
ts-hosts-alllocate-Executive and propagated by the planner after
recent patches. It contains $flight.$job.
Jobs which do not include this are anonymous and are listed
separately, using the resource name and info field (if present) as the
job name.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Mon, 5 Oct 2015 14:10:36 +0000 (15:10 +0100)]
standalone: Use fail() from mgi-common in most places
Functional change is simply to prepend "$0: ", to change the exit
code for unknown operation and to slightly alter the error message
when no arguments are given.
A few "exit 0" and "exit $rc" remain.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Fri, 22 May 2015 11:56:15 +0000 (12:56 +0100)]
standalone: Make it possible to pass options to run-test
Currently the remainder of the comnand line is passed after the host=
ident, which allows for other idents to be given, which isn't all that
useful in practice.
Instead arrange that any additional options up to a "--" marker are
passed before host= and anything after are passed after.
Since the options themselves have a leading -- this can confuse the
scripts own option parsing, meaning you may need more than one "--"
marker, the first to separate the standalone helper args from the ts
args and a second to separate from any ident optiopns.
Ian Campbell [Fri, 25 Sep 2015 13:06:13 +0000 (14:06 +0100)]
cs-adjust-flight: Add job-status to report job stats
The return code of sg-run-job does not reflect the state of the job,
which is instead written to the database. For the benefit of running
tests in a loop until failure add a command to retrieve the status to
stdout.
Add a get-job-status command to the standalone helper script.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 29 Sep 2015 10:24:53 +0000 (11:24 +0100)]
production-config: Switch DebianMirrorProxy to port 3143
This is running apt-cacher-ng rather than apt-cacher (which remains on
port 3142 for the time being). It seems that apt-cacher exposes a bug
(#795284) in Jessie's version of apt.
apt-cacher-ng also seems more popular these days.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 29 Sep 2015 09:44:53 +0000 (10:44 +0100)]
Debian: Uninstall flash-kernel when creating our own boot.scr
flash-kernel will run from various kernel postinst hooks and overwrite
our own boot scripts. While this might be tollerable for the initial
installation we don't want to risk it occuring after we have created
our own boot.scr to boot xen.
dpkg --purge succeeds if the package wasn't installed.
This happened to show up with Jessie since it now supports the two
boards in our test lab while Wheezy didn't (so flash-kernel didn't
know about them and did nothing).
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 30 Sep 2015 15:02:49 +0000 (16:02 +0100)]
Executive HTML output: Use #888888 (grey) for queued jobs
Either the flight hasn't started yet, or the job is blocked waiting
for other jobs to finish. In any case their state is not very
interesting.
Most usefully this change visually distinguishes, in the plan summary,
jobs which are waiting for prior jobs to finish, from ones which have
entered the planning queue.
Also replace a $f->{status} with $status, which is less confusing.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Get test right.
Update commit message.
Ian Jackson [Thu, 1 Oct 2015 10:55:47 +0000 (11:55 +0100)]
ts-logs-capture: Actually do hard host reboot sometimes
The logic in try_fetch_logs for setting $ok was wrong. $ok would be
set if we reached the end of any outer (pattern) loop iteration. If
the host is actually dead all the pattern expansions would fail, but
some of the patterns are literals and do not need expansion. The
inner (logfile) loop would say `next' if the logfile fetch failed, but
that just goes onto the next logfile. So this code would always set
$ok.
Instead, set $ok to 1 when we successfuly fetch any logfile or
successfully expanded any pattern (even if it didn't match any files).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 30 Sep 2015 12:01:40 +0000 (13:01 +0100)]
make-flight: Trim the matrix of disk format flights
We don't need to test every combination of toolstack, architecture,
and disk format. We don't expect many architecture-specific bugs in
the per-disk-format code in the toolstack layers.
We _do_ want to test every combination of toolstack and disk format
(since the format configuration machinery is toolstack specific) and a
reasonable selection of architectures for each disk format (since
arch-specific bugs in actual underlying disk drivers are a
possibility).
The implementation strategy is for do_pv_debian_tests to select a
particular architecture for each combination of toolstack and format.
(Because the architecture is actually in an outer loop, we recalculate
that selection multiple times, and skip inner iterations for the other
architectures. This is all in bash code so the wasted computation is
not particularly important.)
We have a safety catch which spots if any architecture is entirely
untested in any of these combinations; this would happen if a new
architecture is introduced elsewhere and not added to the list. We do
not have a safety catch which spots when a (toolstack,format)
combination becomes untested due to deletion of an architecture.
(That would be more fiddly to implement without restructuring.)
We list armhf twice because we would like to do at least as many ARM
as x86 tests (particularly given our current workload and capacity).
The result is that the set of generated jobs is adjusted as follows:
Ian Jackson [Tue, 29 Sep 2015 14:25:47 +0000 (15:25 +0100)]
ts-hosts-allocate-Executive: Finish a couple of transactions
Call $equivflagscheckq->finish() at the end, rather than in the
candidate search loop. This avoids missing a `next' control path.
Call $resprop_q->finish() at all.
One of these is responsible for this message:
DBI::db=HASH(0x88e79c4)->disconnect invalidates 1 active statement
handle (either destroy statement handles or call finish on them
before disconnecting) at Osstest/Executive.pm line 708, <GEN4> line
53.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Tue, 29 Sep 2015 13:47:33 +0000 (14:47 +0100)]
ts-hosts-allocate-Executive: Do not allocate specific host with wrong blessing
If the job contains a runvar specifying a specific host, still check
that host's blessing. Otherwise bisections can run on unblessed
hosts. They should fail instead.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Tue, 29 Sep 2015 13:19:05 +0000 (14:19 +0100)]
Do not multiply console hvc0 getty entries
target_kernkind_console_inittab is supposed to edit inittab to make
sure that there is a getty running on hvc0.
However:
- It is not idempotent; if run more than once it can produce duplicate
entries `1:' and `xc:'.
- It works by copying and editing the entry `1:' but it might be that
there is already another console entry for hvc0 for some other
reason.
If we end up with multiple entries for hvc0, we can have two copies of
getty fighting, and if you manage to log in, one of them will be
fighting with your shell.
Guard the script with a grep, which looks for inittab entries
mentioning the intended console. This makes makes it do nothing if
nothing is needed (and therefore it also makes it idempotent).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 28 Sep 2015 15:29:33 +0000 (16:29 +0100)]
TestSupport: Honour $stdin fh argument to cmd, tcmd and tcmdex
These are internal functions, so the change is entirely within
TestSupport. All the call sites are adjusted to pass undef so there
is no functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
(See the comment in the new file for the explanation.)
This change affects all our Debian installs (both hosts and guests)
which are done with preseeding, because preseed_base() arranges to
install overlay/.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Thu, 24 Sep 2015 17:03:00 +0000 (18:03 +0100)]
Shell fixup: Use bash in posix mode
When bash is started as /bin/sh it run in posix compatibility mode.
But when invoked as /bin/bash it does some things ... differently.
Most notably:
Subshells spawned to execute command substitutions inherit the
value of the -e option from the parent shell. When not in posix
mode, bash clears the -e option in such subshells.
It is a mystery why anyone thought the `non-posix' behaviour was
desirable. One effect in practice is that osstest's cr-daily-branch
can blunder on if one of its version fetches fails.
AFAICT the only documented way to get rid of this anomalous behaviour
is to switch bash to posix mode. I have read through the wheezy
bash(1) manpage and searched for posix, and the following behavioural
differences are described:
* Differences in interative startup (not relevant to us).
* Minor (irrelevant) differences in which startup files are read
during noninteractive startup. (Eg, BASH_ENV not honoured.)
* Differences to the parsing of invocations of `time'
* `test a == b' may be unsupported (but it's wrong and we say `=')
* -e not inherited by some subshells (this is what I am trying to fix)
* `.' and `source' do not search the cwd.
* `set' with no arguments does not print shell functions etc.
So I think, with the previous patch, that these changes are all
desirable or at least harmless.
I have not added `set -o posix' to shell script fragments invoked by
various scripts (eg Perl and Tcl scripts). Those scripts might be
processed by bash if /bin/sh is bash, but when is invoked as sh it
runs in posix mode anyway.
I have done some ad-hoc testing but it seems like much of this is
difficult to test. I suggest we push it at a time when we can keep a
close eye on the behaviour.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 24 Sep 2015 15:46:44 +0000 (16:46 +0100)]
Shell fixup: Make all invocations of `.' (`source') use ./
In POSIX, `.' (the shell builtin) respects PATH, and does not search
`.' (the current directory).
Change all the invocations which refer to files which are part of
osstest to say `. ./foo' instead of simply `. foo'.
I have checked the results of
git-grep '^[ \t]*\. [^./]'
after this patch and the remaining five hits are of no concern.
As a double-check of my hand-editing, I have also done this
perl -i~ -pe 's#^(\s*\. )\./#$1#' *
and verified that the resulting tree is almost identical to that
before this commit. There is one difference, where the original
code already said `. ./job'.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Campbell [Fri, 25 Sep 2015 10:21:08 +0000 (11:21 +0100)]
Do not run baseline tests on osstest branch.
If an osstest instance is running flights with
OSSTEST_BASELINES_ONLY=y (e.g. on a secondary site, like the Citrix
Cambridge instance) then it will also be running a regular osstest
flight to merge from the upstream osstest and does not want to also do
baseline testing.
I expect this will be the normal configuration in all sites other than
the master colo production instance, so arrange for it via
branch-settings.osstest rather than some site local configuration file.
In the colo production instance we never set OSSTEST_BASELINES_ONLY=y,
so this has no effect. If we did set that option I think suppressing it
for the osstest branch would still be correct.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Wed, 23 Sep 2015 15:48:36 +0000 (16:48 +0100)]
Debian: Wait for udev devices to settle in erase-other-disks
Otherwise we, apparently, have a race between the "test -b" + dd
against the sda1 device node being removed and end up creating a file
called /dev/sda1 containing 32K of zeroes, which will later render
mke2fs very confused.
For some reason we seem to reliably loose the race on Jessie i386 host
installs, but not on Wheezy or other arches on Jessie.
This was tricky to diagnose because the Debian installer main-menu
process appears to buffer the output of things it runs before logging
them to /var/log/syslog, which means the output of "set -x" and the
actual affect of the commands (e.g. in the kernel messages) can be
presented in very confusing orders. Hence this patch also adds
extensive use of logger(1) to record what it is actually doing at the
time it does it. These logs include the pid of the command too since
main-menu only logs its own pid, not that of the subcommand which
gneerated the output.
Lastly this patch adds an explicit test that each device it touches is
actually still a block device afterwards.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Mon, 21 Sep 2015 15:14:33 +0000 (16:14 +0100)]
Debian: handle move of ip(8) to /sbin in Jessie
Unfortunately udev treats non-absolute commands as relative to
/lib/udev rather than consulting $PATH, so we have to figure out the
path based on the suite.
Without this the force-mac-address workaround (needed on Arndale)
doesn't work.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 25 Sep 2015 11:09:09 +0000 (12:09 +0100)]
Host install: Break out target_core_dump_setup
We are going to want to do this for nested HVM L1s too.
No functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v14: Split this nfc patch out from other changes
Drop changes to whitespace usage
Rename new function from `core_dump_setup'
Drop introduction of an unnecessary mkdir -p
Robert Ho [Mon, 17 Aug 2015 05:48:04 +0000 (13:48 +0800)]
Debian HVM guests: Honour guest disk and ram size runvars
Allow runvars to specify guest disk and ram size, turning previous
values into defaults:
The default disk size for the guest is `10000M' which is not going to
be sufficient for nested HVM tests. We are going to want to use a
larger disk size for the nested L1. The appropriate disk_size will be
defined in make-flight, in forthcoming changes.
Also, also allow ram size to be defined by runvar. The runvar takes
precedence over the default (which is calculated based on host RAM
etc.)
No functional change with current sets of runvars.
Signed-off-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Robert Ho [Mon, 17 Aug 2015 05:52:35 +0000 (13:52 +0800)]
Debian HVM guests: Comment out CDROM entry in HVM guest VM
Comment out the CDROM entry in the installed system's sources.list.
The installation ISO is not generally present in the virtual CDROM
while the guest is running, so this entry is nonfunctional.
Removing it causes the L1 HVM guest VM to use the http entry instead.
The result is that now `apt-get' works in the guest.
Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Robert Ho [Mon, 17 Aug 2015 03:34:04 +0000 (11:34 +0800)]
Debian grub2: Optimize and re-format submenu parsing
* space between ')' and '{'; and after '='
* omit unnecessary 'define' and '!defined' usage
* break long '{}' into several lines
Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v14: Drop removal of MenuEntryPath setting in grub2 submenu parse
Ian Jackson [Mon, 21 Sep 2015 16:49:48 +0000 (17:49 +0100)]
Executive: Delay releasing build host shares by 90s
When a build job finishes, the same flight may well want to do a
subsequent build that depended on the first. When this happens, we
have a race:
One the one hand, we have the flight: after sg-run-job exits,
sg-execute-flight needs to double-check the job status, and search the
flight for more jobs to run; it will spawn ts-allocate-hosts-Executive
for the new job, which needs to get its head together, parse its
arguments, become a client of the queue daemon, and ask to be put in
the queue.
On the other hand, we have the planning system: currently, as soon as
sg-run-job exits, the connection to the ownerdaemon closes. The
ownerdaemon tells the queue daemon, and the planning queue is
restarted. It might even happen that coincidentally the planning
queue is about to start.
If the planning system wins the race, another job will pick up the
newly-freed resource. Often this will mean unsharing the build host,
which is very wasteful if the releasing flight hasn't finished its
builds for that architecture: it means that the next build job needs
to regroove a host for builds.
Add a bodge to try to make the race go the other way: after a build
job completes successfuly, do not give up the share for a further 90
seconds. (We have to use setsid because sg-execute-flight kills the
process group to clean up stray processes, which this sleep definitely
is.)
A better solution would be to move the wait-for-referenced-job logic
from sg-execute-flight to ts-hosts-allocate-*. But that would be much
more complicated.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: New patch
The Debian i386 image boots a 32-bit non-PAE kernel which therefore
cannot have any PV drivers on our 64-bit hypervisors. This makes it a
bit slow: in my test on a machine under my desk it took 1400s out of
the allowed 2000s.
With this change the timeout is 3000s instead.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
We look up a guest runvar GUEST_CONTEXT_timeoutfactor (according to
the usual rules for guest runvars).
Here CONTEXT can currently be `general' or `install'. (`install'
applies when the guest is being installed.) If the runvar exists, all
timeouts relating to that guest in that context are increased by the
specified factor.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 18 Sep 2015 16:23:39 +0000 (17:23 +0100)]
Timeouts: Introduce target_adjust_timeout
This function is now called for almost every timeout.
Specifically, it is called in the cases in TestSupport where a guest-
or host-related timeout is passed from code which has a $ho or $gho,
to code which does not: all callers of poll_loop, and tcmd.
Currently the function is a no-op. Its existence and use will allow
us to introduce various provocations for adjusting timeouts, of which
I have one planned.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 18 Sep 2015 15:24:24 +0000 (16:24 +0100)]
ts-debian-hvm-install: Use /dev/sda for i386, not /dev/xvda
The device shows up as /dev/sda in the installer. This is because the
i386 installer image (in wheezy and jessie, at least) is not PAE, and
therefore does not load PVHVM drivers and therefore does not switch
from the emulated device (which shows up as /dev/sda) to blockfront
(/dev/xvda).
There are ways it might be possible to provide an i386 install image
with PVHVM drivers (for example, the Xen-enabled i386 kernel on the
multi-arch ISO). However it is useful to have a test job which tests
both emulated devices and PVHVM ones, within a single job, and even on
my ancient under-desk test box (bedbug) the whole test takes only 28
minutes, which seems fine even as part of the fast smoke test.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 18 Sep 2015 15:21:45 +0000 (16:21 +0100)]
ts-debian-hvm-install: Do not create EFI partition if EFI not in use
If we are booting our install ISO using a non-EFI executable, don't
try to provide an EFI for the installed system either.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Fix regexp to match efi.img (!)
Rebased, fixed code motion conflict.
Ian Jackson [Fri, 18 Sep 2015 14:35:47 +0000 (15:35 +0100)]
ts-debian-hvm-install: Set $gsuite after $gho
$gsuite was set from guest_var, but before $gho was set, leading to an
undefined value warning from Perl.
This would ignore any guest-specific suite runvars. AFAICT these are
set by some of the jobs in make-distros-flight. I think the effect of
this change is to apply workarounds for the intended suite, rather
than for wheezy.
(Although there is another assignment to $gho later in
ts-debian-hvm-install, for stage 2, the stage 2 code does some trivial
TestSupport calls and does not need $gsuite. So there is no need to
make arrangements to assign to $gsuite - or, for that matter, $kernel
or $ramdisk, in that path.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 18 Sep 2015 14:30:57 +0000 (15:30 +0100)]
ts-debian-hvm-install: Cope with images containing only isolinux
debian-7.2.0-i386-CD-1.iso contains no grub, only isolinux.
If the specified EFI grub file does not exist, fall back to isolinux.
This requires a -c option as well, according to
https://wiki.debian.org/DebianInstaller/Modify/CD
Only try to set up a grub config if we are booting grub. (The i386
image in question does not contain a [debian]/boot/grub directory.)
If boot/grub/efi.img _does_ exist (ie, for other existing tests), the
only difference in behaviour is to reorder slightly the options to
genisoimage: `-b boot/grub/efi.img' now occurs after `-no-emul-boot
-r' rather than before.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Log $bootfile value.
Preseed generation now happens later due to previous patch;
so $bootfile setting now also deferred. Context change only.
Ian Jackson [Mon, 21 Sep 2015 14:13:23 +0000 (15:13 +0100)]
ts-debian-hvm-install: Defer preseed generation
Defer preseed file generation until after we have fetched and looked
inside the install image, because we are going to want to make changes
to the preseed file based on the image contents.
No overall functional change, although some things happen in a
different order now, and the ISO manipulation takes place in two calls
to target_cmd_root rather than one.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: New patch. Needed because otherwise the test for the grub install
image (to be introduced in the next patch) happens before the ISO
is unpacked, and we would then always fall back to isolinux.
Ian Jackson [Fri, 18 Sep 2015 13:57:47 +0000 (14:57 +0100)]
ts-debian-hvm-install, etc.: Do not hardcode in-iso path
ts-debian-hvm-install hardcoded `install.amd' as the directory in the
.iso in which to find the kernel and initrd. This is wrong for
architectures other than amd64.
Instead, pass this information in runvars (as is done for the netinst
installs in make-distros-flight), and honour it in
ts-debian-hvm-install.
If the runvars are not set, default to the previous hardcoded values.
(This arranges that clones of old flights still work with new osstest,
eg for bisection.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 8 Jul 2015 15:21:05 +0000 (16:21 +0100)]
ap-*: Be able to fetch and push xen.git#smoke
The branches and push gates are now:
xen.git#staging -[xen-unstable-smoke]-> #smoke -[xen-unstable]-> #master
Deployment note: When this passes the osstest self-push-gate, the main
xen-unstable flight will start using smoke as an input. Therefore,
until the new cronjob is installed to run the xen-unstable-smoke
tests, an automatic process should keep xen.git#smoke up to date with
xen.git#staging. Eg, running in screen in xen@xenbits:~/git/xen.git:
while sleep 1800; do git fetch . staging:smoke; done
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: `xen.git#smoked' branch name changed to `#smoke'
Ian Jackson [Thu, 17 Sep 2015 15:17:25 +0000 (16:17 +0100)]
cr-daily-branch: Use mg-adjust-flight-makexrefs to have smoke tests reuse builds
The smoke tests are for testing xen-unstable. We want to avoid
building anything else. So arrange to reuse previous builds by
calling mg-adjust-flight-makexrefs.
We rebuild libvirt too. This is necessary because libvirt is built
against xen.git, and uses ABI-unstable APIs, so we need a libvirt
built against the right xen.git. This means, for the smoke tests, we
need to build libvirt ourselves. Currently this build seems to take
416 sends (from host allocation, which we - perhaps naively - hope
will be able to reuse the host from the just-finished build job).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Keep build-amd64-libvirt too.
v3: Add a comment about the --blessings=real
v2: New patch
Ian Jackson [Fri, 3 Jul 2015 18:56:45 +0000 (19:56 +0100)]
Provide xen-unstable-smoke branch
Introduce support for branch=qemu-xen-unstable-smoke which has
xenbranch=xen-unstable-smoke.
In make-flight, this contains a very limited set of jobs
test-amd64-amd64-libvirt
test-amd64-amd64-xl-qemuu-debianhvm-i386
test-armhf-armhf-xl
and the builds they depend on.
The debianhvm job exists only in this flight, and is generated by
having branch_debianhvm_arch return i386 instead of amd64. This is so
that this branch contains a 32-bit x86 guest as well as a 64-bit one.
We override host allocator parameters to make this flight not care
about host stickiness: it just takes whatever comes to hand. These
runvars are marked `synth' so that cs-bisection-step and
cs-adjust-flight do not copy them, as discussed in previous patches.
Later we will arrange to reuse previous builds for the build artefacts
which aren't intended subjects of the smoke test.
(Deployment note: This needs images/debian-7.2.0-i386-CD-1.iso which I
have already placed in the Cambridge and Xen Project instances.)
In ap-common we need to arrange to use the same qemu trees as for
xen-unstable, rather than looking for special smoke ones.
In select_xenbranch xen-unstable-smoke is mostly like xen-unstable.
There are only two places in osstest where xenbranch `xen-unstable' is
treated specially and only one of them needs adjusting to match
xen-unstable-smoke too.
The new branch `xen-unstable-smoke' has a `prev' branch of
`xen-unstable' according to cri-getprevxenbranch, which is technically
wrong, but this is not important because xen-unstable-smoke has no
prev tests.
We are going to sort out the push gate ref plumbing in xen.git in the
next osstest patch.
Also, use a branch-settings file to set the new branch's resource
priority to -20 to make it run ahead of anything else automatic.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Introduce xenbranch_forqemu into ap-common.
Combine patches for make-flight and cr-*. make-flight includes
cri-common and ap-common (which is arguably a layering violation,
but there we are).
Dropped ack from Ian Campbell.
Set `qemuubranch' correctly in select_xenbranch.
v2: Generate all the jobs that this flight's tests use, and add
note about this to the commit message.
Mention `synth'-ness of hostalloc runvars in commit message.
Image is in Xen Project test colo too.
Ian Jackson [Fri, 3 Jul 2015 18:55:32 +0000 (19:55 +0100)]
make-flight: Run job_create_test_filter_callback on true job name
job_create_test would pass $job to job_create_test_filter_callback but
then later maybe append -xsm to it. Fix this.
No functional change for existing in-tree code because all existing
tests of the $job end in *.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix textual conflict after dropping "make-flight: Allow separate
specification of pre-built Xen vs others"
Ian Jackson [Thu, 17 Sep 2015 12:43:17 +0000 (13:43 +0100)]
cs-adjust-flight: Provide `jobs-del' operation
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Document the new operation
v2: New patch
Ian Jackson [Wed, 16 Sep 2015 15:48:05 +0000 (16:48 +0100)]
sg-check-tested: New --pass-job= option
Specifies that returned information should relate to a flight in which
a particular job existed and passed. The option can be repeated if
desired (to specify flights in which _all_ those jobs passed).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch
Ian Jackson [Mon, 21 Sep 2015 13:35:15 +0000 (13:35 +0000)]
sg-report-flight: Better searching for used revisions
The old algorithm used for determining which flight might be a
suitable test of a particular revision was rather crude, in two ways:
* It would look at _all_ jobs in a flight referred to from the flight
of interest, not just at the relevant jobs;
* It would only look at the direct referents of the flight in
question. So for example, if a flight of interest contained
test-amd64-i386-libvirt, it would find a referenced
build-i386-libvirt in another flight, but that build refers to
build-i386, and it would not look at that (unless it happened to be
in the same flight).
Fix this by redoing the revision archaeology, with some $why tracking
to explain how we found a particular revision.
cs-bisection-step and sg-check-tested arguably ought to do do it this
way too. But I am leaving centralising this new logic, and using it
in those other programs, for another day.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: New patch
Ian Jackson [Thu, 2 Jul 2015 17:27:28 +0000 (18:27 +0100)]
ts-host-alloc-Executive: Honour various hostalloc_* runvars
We honour
hostalloc_maxbonus_variation
hostalloc_bonus_previousfail
hostalloc_bonus_sharereuse
and make them default to their previous values.
These should be set as `synth' runvars during flight construction, so
that they are not copied into flights generated by cs-bisection-step
or cs-adjust-flight. cs-bisection-step makes its own arrangements for
host specification. So should the caller of cs-adjust-flight (perhaps
via the blessing system).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Incorporate note about `synth'-ness in commit message.
Ian Jackson [Wed, 16 Sep 2015 12:14:47 +0000 (13:14 +0100)]
standalone-generate-dump-flight-runvars: Show synth runvars
Pass -a to mg-show-flight-runvars. That way when we use the new
cs-job-create feature to set synth runvars during creation, we will
see them in the dump.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 16 Sep 2015 12:09:01 +0000 (13:09 +0100)]
mg-show-flight-runvars: Decorate synth runvar names with ~
Make mg-show-flight-runvars -a append ~ to the names of synth runvars.
(This is consistent with the new syntax in cs-job-create.)
We do this by editing $row[1] (and $colws[1]) so we can avoid
disturbing the general column format calculation and printing.
We switch to fetchrow_array rather than fetchrow_arrayref. This is
clearer and also avoids having to copy $row (because the value in the
DB $row from fetchrow_hashref would be readonly).
We have to check for $synth eq 'f' as well as $synth being boolean
false, because SQLite's typeless nature (or, to put it another way,
DBD::SQLite's failure to look at the schema) means that a boolean
field's value of 'f' or 't' is simply returned as a string to Perl.
But of course "f" is trueish in Perl.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Use fetchrow_array instead.
Do not mistakenly drop the $synthcond assignment (!)
Ian Jackson [Thu, 2 Jul 2015 17:33:19 +0000 (18:33 +0100)]
cs-job-create: Permit creation of `synth' runvars
This will be useful for some hostalloc_* runvars which we are going to
introduce shortly.
This is going to be the way to set a runvar which is not copied by
cs-bisection-step or cs-adjust-flight. Using `synth' for this is
arguably slightly wrong but it does the right thing in all existing
cases. The alternative would be a schema change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Defer some of the discussion to later commit messages.
projects / people / liuw / osstest.git / log