Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> CC: Marcos Matsunaga <Marcos.Matsunaga@oracle.com>
Ian Jackson [Tue, 18 Oct 2016 15:46:20 +0000 (16:46 +0100)]
mgi-common: Support empty (unset) HttpProxy properly
mg_update_proxy ends up being set to the empty string so the {...:+-x}
form is needed to expand only non-empty values to `-x'.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Reported-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Marcos Matsunaga <Marcos.Matsunaga@oracle.com>
Ian Jackson [Tue, 18 Oct 2016 15:22:33 +0000 (16:22 +0100)]
standlone-reset: mkdir some directories
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Marcos Matsunaga <Marcos.Matsunaga@oracle.com>
Ian Jackson [Tue, 18 Oct 2016 14:50:16 +0000 (15:50 +0100)]
standalone-reset: Fix when TftpDiVersion not set in config
In b8134c7fa60d "mg-debian-installer-update: Print the correct value
for TftpDiVersion", the output of mg-debian-installer-update was
changed to be a config fragment. But standalone-reset expected it to
be just the date value, and was not updated.
Update it now. And leave a comment in mg-debian-installer-update to
stop this happening again.
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Marcos Matsunaga <Marcos.Matsunaga@oracle.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 14 Oct 2016 14:33:03 +0000 (15:33 +0100)]
support check: Reverse sense of return values
The toolstack()->check_blah functions would return an exit status.
This is very confusing. Instead, have them return a booleanish value
representing the support status: ie, truthy if supported.
No functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 14 Oct 2016 14:24:45 +0000 (15:24 +0100)]
support check: Provide helper function to print nice log message
Makes ts-migrate-support-check and ts-saverestore-support-check
slightly clearer.
This function takes $yes, which is truthish if the feature is
supported. We are going to replace use of exit status truth values in
the various check functions in just a moment.
No functional change other than to log output.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 6 Oct 2016 16:38:29 +0000 (17:38 +0100)]
libvirt: Check /capabilities/host/migration_features/live for live migration
libvirt is capable of advertising this separately from
/capabilities/host/migration_features, so if save/restore is supported
but live migration is not, this will do the right thing.
We would have preferred libvirt to advertise
/capabilities/host/migration_features/save
or something, but it doesn't right now, so we continue to use
/capabilities/host/migration_features
to detect save/restore support.
If libvirt changes its feature presentation, then at some future point
we should change osstest too.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Martin Kletzander <mkletzan@redhat.com> CC: Jim Fehlig <jfehlig@suse.com>
---
v3: Call correct function name.
Ian Jackson [Tue, 4 Oct 2016 16:24:17 +0000 (17:24 +0100)]
libvirt: Do not attempt save/restore when migration not advertised
Currently, osstest wrongly thinks that ARM can do save/restore,
because `virsh help' does mention the save command (on all
architectures).
So, additionally, check the virth capabilities xpath
/capabilities/host/migration_features
to try to see whether this host supports migration.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Julien Grall <julien.grall@arm.com> CC: Jim Fehlig <jfehlig@suse.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v3: Removed questioning and uncertain remarks from commit message,
following appropriate confirmation from libvirt folks.
Get sense of conditional combination right.
Ian Jackson [Tue, 4 Oct 2016 16:15:55 +0000 (17:15 +0100)]
libvirt: Check migration capabilities using proper XML parser
Do not grep the virsh capabilities output (!) Instead, parse the XML
using perl's XML modules and look for the specific feature flag using
an XPATH pattern.
Xen could in principle (and is expected to, in the future, on ARM)
support save/restore but not live migration. Currently it supports
neither on ARM. libvirt's capabilities system does in principle
capture this distinction, but only in an adhoc way.
For now, this osstest commit has no ultimate functional change (with
libvirt output as it currently appears on our real hosts).
Deployment note: Requires libxml-libxml-perl to be installed.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Julien Grall <julien.grall@arm.com> CC: Jim Fehlig <jfehlig@suse.com>
---
v3: Mention newly-required Perl libraries in README and commit message
Get answers and syntax right (!)
Ian Jackson [Thu, 6 Oct 2016 18:39:02 +0000 (19:39 +0100)]
make-flight: XTF: honour $bfi (ie build flight)
If make-flight is run with a $buildflight argument, it does not create
any build jobs. The test jobs are supposed to refer to the build jobs.
This was not done correctly for the XTF tests. Add the missing ${bfi}.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Reviewed-by: Wei Liu <wei.liu2@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Wed, 5 Oct 2016 13:39:11 +0000 (14:39 +0100)]
Support guest-specific "toolstack" for guest creation
Some guests need creation in a special way. For example, rump kernels
are ideally started with rumprun. Honour a guest var which specifies
a toolstack name.
Osstest::TestSupport::toolstack now takes an optional $gho so it can
do this lookup when appropriate.
After creation the guest is necessarily managed with the toolstack for
the host, so we honour this (ie we pass the $gho) only for create.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Sep 2016 12:57:05 +0000 (13:57 +0100)]
rump-test-net: setsockopt V6ONLY off
NetBSD (unlike Linux) has the V6ONLY socket option turned on by
default. So to work in the rump kernel environment when tested with
IPv4 we need to adjust this setting.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Sep 2016 10:36:23 +0000 (11:36 +0100)]
rump-test-net: New test program
The rump kernel WOPR test is no more, so we reimplement it. This test
program simply listens on a TCP socket and says hi when you connect to
it. It's a portable program. So far, this has been tested on Linux,
but not in the rump environment.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 6 Oct 2016 15:49:28 +0000 (16:49 +0100)]
mg-allocate: Provide command line way to list allocated resources
Freely shareable resources don't appear in the plan, and the plan is
not always immediately updated, and is generally not always a
convenient interface. Provide a command line way to list allocated
resources.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 6 Oct 2016 12:10:20 +0000 (13:10 +0100)]
sg-report-flight: Avoid some warnings when reporting unexecuted jobs
If no steps in a job are executed, there can be a failure with a
synthetic step row, containing a stepno of ''. This causes a perl
warning when compared with <=>:
Argument "" isn't numeric in numeric comparison (<=>) at ./sg-report-flight line 774.
Fix this by replacing falseish values with 0.
Bug introduced in 0e09a8b00ec6 "sg-report-flight: Report earlier,
earlier step failures".
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 4 Oct 2016 18:09:18 +0000 (18:09 +0000)]
host allocation: Fix duration estimate to not include host allocation
In 720f08cb9052 "Executive: Previous duration estimator: use overall
time, not sum of steps" we introduced a bug: the condition to exclude
the host allocation time is now not effective if there are any steps
before host allocation. Usually there are.
This means that the host allocation duration estimator has been
including the host allocation time from previous jobs, which is quite
wrong.
Fix this by subtracting the maximum duration of any host allocation
step. Hopefully there will only be one.
If any host allocation runs concurrently with other steps (including
other host allocations) then this will start to give wrong answers.
But there are other reasons why we wouldn't want to do that.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Fix sql syntax.
Ian Jackson [Tue, 4 Oct 2016 17:20:51 +0000 (17:20 +0000)]
host allocation: Support -B<blessing> option
The flight's intended affects the hostflags required, the duration
searches, and other decisions. It is particularly useful for
debugging, where it can be desirable to try replaying a production
job's allocation with a "play" job.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 4 Oct 2016 12:10:33 +0000 (13:10 +0100)]
flight preservation: Honour flight allocation during expiry
Look in the resources and tasks table for a resources table entry
corresponding to each flight, owned by a live task. Such flights are
not deleted.
Specifically:
* At the start, we get a list of all the preserved flights, and
also print the information to stdout.
* Whenever we compare flight numbers for inequality (as a proxy
for flight age), we first compare where the flights are allocated.
(When there are references, the effect is that an allocated
referring flight counts as very late, so $latestref will contain it.)
* Before actually deleting the selected flight we check it's not
allocated. (Strictly, we check it's "latest" reference is not
allocated.)
Currently there is nothing which creates such resources table entries
so there is no overall functional change. Also, as a result, the doc
reads rather oddly. This will be fixed in the next patch.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 4 Oct 2016 12:20:12 +0000 (13:20 +0100)]
mg-allocate: Tiny refactoring
Break out $shareix assignment from $4. (We are going to want to put
some code just after this point which will want to do regexp matching,
which would trash $4.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 4 Oct 2016 12:07:37 +0000 (13:07 +0100)]
cr-ensure-disk-space: -F option
May be repeated (cuddled with itself) or given a number. Forces
deletion, even if there is enough space. Normally clean up one less
flight than specified, since cr-ensure-disk-space reruns its check
after acquiring the lock.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Wei Liu [Thu, 8 Sep 2016 15:52:48 +0000 (16:52 +0100)]
TestSupport: use qemu-img to create vhd image
We would like to delete blktap2 from xen.git at some point, but vhd-util
is part of blktap2. Let's switch to use qemu-img to create vhd image to
remove the dependency on blktap2 in osstest.
We want to use the distro qemu-utils, so add that to the package list
in ts-xen-install.
Note that vhd format is named "vpc" in qemu-img.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 20 Sep 2016 17:46:26 +0000 (18:46 +0100)]
mg-* update: Honour HttpProxy
We need to use the cache in Massachusetts because of the new strict
firewall.
The code in the fetch subtroutine is a bit fiddly because 1. we want
to memoise the config lookup and 2. fetch can be called in a different
directory, so we need to cd back.
While we're here, document why we use the no-cache pragma.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Wei Liu [Tue, 26 Jul 2016 11:16:29 +0000 (12:16 +0100)]
Introduce ts-xtf-run
This is the main script for running XTF. It will first perform
selftest, and then run each XTF test case as a substep.
It does the following things:
1. Run self tests for individual environment and record the result.
2. Collect tests according to available environments.
3. Run the collected tests one by one.
The script may exit early if it detects the test host is down or
xtf-runner returns non-recognisable exit code.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Wei Liu [Thu, 21 Jul 2016 14:37:48 +0000 (15:37 +0100)]
ts-xen-build: always compile in FEP support
By default FEP depends on debug flag. When we are near release the debug
flag will be turned off. In order to test a release build, we explicitly
enable FEP in build configuration.
Since we target Xen versions that already have Kconfig support, only a
Kconfig option is created for now.
We can easily add config option for older Xen when necessary.
Note that this only compiles in FEP support. To enable it a user needs
to explicitly specify fep=1 in hypervisor command line.
Signed-off-by: Wei Liu <wei.liu2@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 7 Jul 2016 18:35:12 +0000 (19:35 +0100)]
Executive: Support substeps
ts-* scripts can now create `substeps'. For the purposes of
archaeology etc., a substep is just like a step. But it does
correspond to a single specific ts-* invocation.
Instead, it is started and finished explicitly as required.
The whole job implementation code needs to explicitly assign a unique
stable testid to each substep.
The `script' parameter is stored in the `step' field in the database,
which is used only for reporting. These do not need to be unique.
All substeps started are should also be finished, by the end of the
job. If this is not done, the job will be regarded as broken (if it
is not already failed or aborted). (But a substep might be finished
by a different ts-* script to the one that started it.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Wei Liu <wei.liu2@citrix.com>
Ian Jackson [Wed, 6 Jul 2016 14:22:21 +0000 (15:22 +0100)]
ts-hosts-allocate-Executive: Support diverse-CLASS hostflag
Specifically:
* Parse it out of the hostflags when constructing the hid
* Look for the `hostalloc-diverse-FLIGHT-CLASS' ClientNote in
the resource plan, to avoid inappropriately planning to reuse hosts.
* Look for the `diversehosts_CLASS' runvar in other jobs in this flight,
to find out who might have allocated with the same CLASS. (This
sort of duplicates information in *hostflags and *host, but digging
the information out of the latter two would be very tiresome.)
* Check each of the above for each candidate host.
* Set the ClientNote when we are preparing a booking.
* Set the runvar when we do the allocation.
* Document the ClientNote.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Wei Liu <wei.liu2@citrix.com>
Ian Jackson [Fri, 8 Jul 2016 19:02:39 +0000 (20:02 +0100)]
step status skip: Implement in sg-report-flight
* When we are doing archaeology, searching for flight(s) which ran a
particular testid, ignore all flights where the testid was skipped.
* In a flight we are examining for failures we need to justify, do not
regard `skip' as a failure which requires investigation. We
thusg treat `skip' in such a flight very like `pass'.
* Assign a colour (dark grey, almost like the background) and display
priority (very low) to `skip', so that they turn up nicely in the
HTML grids.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 8 Jul 2016 18:57:53 +0000 (19:57 +0100)]
step status skip: Ignore in report_run_getinfo
report_run_getinfo is trying to generate some HTML to describe a job's
(current) status. It sometimes looks at the steps to find
`interesting' information to report.
Completely ignore steps with status `skip' for this purpose, just like
we ignore ones with status `pass'.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 8 Jul 2016 18:56:05 +0000 (19:56 +0100)]
step status skip: Ignore in cs-bisection-step
cs-bisection step wants to completely ignore all skipped steps. So we
adjust the one query which doesn't already insist on particular status
values, to filter out `skip'.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 8 Jul 2016 18:30:58 +0000 (19:30 +0100)]
Executive: Previous duration estimator: use overall time, not sum of steps
Some jobs runs steps in parallel. Do not add up all the individual
step durations. Instead, calculate the duration as the time between
first step start and last step finish.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Ian Jackson [Fri, 1 Jul 2016 15:46:08 +0000 (16:46 +0100)]
rumprun: `rumpbake' our executables and run them with `rumprun'
(Well, our one executable: xenstore-ls)
Modern rumprun requires the output of the linker to be `baked' (second
link phase, where the complete unikernel is assembled).
This has to be done as part of the build, because it needs all the
rumpkernel libraries. It generates a single image file - there is no
longer any disk image or config file produced by the rump ecosystem.
The baked file needs to be provided in a dist. We have
ts-rumprun-bake take command line argument specifying which things to
bake. It reads the runvars for the source executables and creates a
single dist output containing the images. There are now `executables'
and `images'.
Furthermore modern rumprun requires the image to be run with
`rumprun'. One underlying reason is that it wants to pass the command
line and some other config parameters to the guest via xenstore, in
/local/domain/GUEST/rumprun/cfg. To do this outside xl requires the
domain to be created paused. Another is to abstract away details of
the actual execution environment (compared to other unikernel
execution models).
rumprun has a mode (-D -T) in which it would be possible to fish the
configuration and the desired json object (for the cfg) out of the
tempfile it creates. It might also be possible for osstest to
construct these out of whole cloth.
However, this would be undesirable because it would break if rumprun
changed (in particular, if the interface to the domain creation
changed).
And because of the cfg wrinkle it still wouldn't let us construct a
domain config file which could be passed to
toolstack($ho)->guest_create.
So instead we invent Osstest::Rumprun::rumprun_guest_create, which
invokes rumprun. rumprun implicitly invokes xl.
The config editing which was previously done by ts-rumprun-demo-setup
is now done by passing appropriate options to rumprun.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 1 Jul 2016 14:44:41 +0000 (15:44 +0100)]
Xen built versions: ts-xen-build: check versions of Xen subtrees, only
ts-xen-build has a check that the actually-built versions of the
various subtrees are right. This allows it to spot if the machinery
for specifying the subtree revision hasn't worked.
However, this machinery is troublesome: it assumes that the value
specified in the revision_TREE runvar is a commit id, just like the
value specified in built_revision_TREE. This is, currently, true in
flights created by cr-daily-branch and cs-try-bisect.
But it is not necessarily true for flights created other ways. In
principle it would be possible to look into each checked out subtree,
and use git-rev-parse (and its equivalent for nother VCSs) to check
whether the specified revision is right (by comparing it to
origin/<revision_TREE>, not <revision_TREE>, I guess). This is quite
fiddly.
The reason this is causing trouble now is that some of the ad-hoc rump
kernel flights I'm currently making contain non-git-revison-id values
for the revision_TREE for parts of the rumprun build.
So for now, limiting this check to TREEs which are actually Xen
subtrees will fix the problem for me (and this will be necessary for
the fuller fix, which I describe above). So do that.
Specifically:
* Add a new WHERE clause to the query statement, so that it selects
only the row for one specific tree
* Run the query once for each tree in %xensubtrees
This leaves the query overly-complicated, but this doesn't matter,
because if and when we make a fuller fix we'll throw this entire query
away. So it is easier to put off rewriting it in the hope that this
will never been needed.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 1 Jul 2016 14:43:00 +0000 (15:43 +0100)]
Xen built versions: Move list of subtrees to BuildSupport
Turn the adhoc list of tree names and subdirectories in
collect_xen_built_versions into a hash, which we iterate over.
Doing this in a data-driven way allows us to provide this information
to callers of collect_xen_built_versions, which is going to be helpful
in a moment.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 30 Jun 2016 14:19:15 +0000 (15:19 +0100)]
rumprun: ts-rumprun-build: Update for newer Xen
Newer Xen needs more work to make it cross compile for rump.
* Pass --host=TARGET to configure. This is needed so that configure
knows that we are deliberately cross compiling. (Otherwise it
tries to run target binaries on the host, and crashes when that fails.)
* Pass CROSS_COMPILE in the environment. This arranges for the Xen
Makefiles to run the right compiler, ie $(CROSS_COMPILE)-gcc.
* Put the rump compiler directory on PATH, so that the Xen Makefiles
can find it.
* Pass HOSTCC=gcc in the environment; otherwise it tries to use the
default CC (which is $(CROSS_COMPILE)gcc), when building
build-system-internal tools which are to be run on the host as part
of the build.
The need for this could be avoided by setting XEN_TARGET_ARCH to the
rump architecture, but then we would have to provide a Xen arch
config file for that architecture, which would be meaningless since
we are not actually building a hypervisor, and would have to contain
various dummy information.
NB in this commit message I use Xen terminology for cross arch names:
Xen GCC/GNU Meaning Example for
terminology terminology rump cross build
host build Native architecture of i586-linux-gnu
the environment in which
we are running the build.
target host Foreign architecture on i486-rumprun-netbsdelf
which the objects etc.
which we are now building
will eventually be run.
n/a target Used only when building a "Canadian"
cross compiler: the 2nd foreign
architecture for which the compiler which
we are now building (on the `build(gnu)'
arch) will, when we run it, produce
binaries (when it is run on the
`host(gnu)' arch).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 1 Jul 2016 17:30:21 +0000 (18:30 +0100)]
Executive: Allow out-of-order manipulations of flights intended play
Flights being operated on by a developer hacking about with the code,
which were created with intended blessing `play', are usually blessed
`running' or `broken' or something. So the safety catch bypass needs
to look at the intended blessing too.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
projects / people / iwj / osstest.git / log