Ian Jackson [Thu, 14 Dec 2017 16:28:21 +0000 (16:28 +0000)]
smoke tests: Fix job reuse globs in cr-daily-branch
A symptom that the old patterns lack !build-arm64-xsm, so the xsm job
might be reused. The overall cause is that it contained a (partial)
list of architectures.
Instead, we observe that:
* The things we want to avoid reusing are Xen and libvirt (which
builds against Xen.
* Non-Xen builds are invariably build-<arch>-<thing> (or even longer)
so we can match them all with build-*-*.
* We need to exclude the Xen builds and the libvirt builds. The
libvirt builds are obvious. The Xen builds are either build-<arch>
which does not match build-*-*, or build-<arch>-xsm.
CC: Stefano Stabellini <sstabellini@kernel.org> Acked-by: Julien Grall <julien.grall@linaro.org> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 14 Dec 2017 16:03:44 +0000 (16:03 +0000)]
smoke tests: Build arm64 with XSM, instead of without
The smoke flight contains test-arm64-arm64-xl-xsm so it should contain
build-arm64-xsm (and not contain build-arm64).
I have checked the results with
OSSTEST_CONFIG=standalone-config-example eatmydata ./standalone-generate-dump-flight-runvars
and looking at the diff shows precisely the expected change.
CC: Stefano Stabellini <sstabellini@kernel.org> Acked-by: Julien Grall <julien.grall@linaro.org> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
This variable can be set to the absolute pathname of a kernel .deb to
use. It will be used only for hosts for which the corresponding
hostflag "need-kernel-deb-<suite>-special" is set.
There is not currently any facility for more than one special kernel
for each architecture.
As with backports kernels, the normal (Debian-supplied) kernel is
available too, and is used for hosts without any special hostflag.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
* In the first half of the backports kernel processing, set
the new variable specialkernel to the string "backports".
(This token occurs in the output .deb and kernel name, and
is also referenced by hostflags of the form
need-kernel-deb-<suite>-backports.)
* Break out the second half of the backports procesing into its
own section, conditional on $specialkernel.
* Replace occurrences of the "backports" token, after that point,
with $specialkernel.
Overall, this will allow other kinds of special kernel to be
introduced. But there is, for now, no functional change.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 14 Dec 2017 13:09:52 +0000 (13:09 +0000)]
di_special_kernel: Use di_installer_path
This avoids the caller having to supply $d_i. This is good because
there is a site we want to call this from which uses that name for a
different value!
No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 14 Dec 2017 12:23:24 +0000 (12:23 +0000)]
di_special_kernel: Replace open-coding in ts-host-install
The effect is simply to reuse the loop in di_special_kernel. The
extra tests etc. to compute $k and $c in di_special_kernel are of no
import here, and are harmless. We have already called
di_special_kernel so if it was going to fail due to this extra
computation, it would do so earlier.
No overall functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 30 Nov 2017 17:40:13 +0000 (17:40 +0000)]
mg-hosts: mknetbootdir: Correctly use Path again
In fcd93edd00a8c9b08da3994d8bee9fa3809c3d93
"netboot: Rework interface to host_netboot_file (was host_pxefile)"
we unintentionally dropped $ho->{Tftp}{Path} from the
effective path. Reinstate it.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 5 Dec 2017 11:47:53 +0000 (11:47 +0000)]
production-config: Increase LogsMinSpaceMby etc.
osstest does not currently try to estimate the prospective disk use in
logs/ etc. of a flight before it starts; nor does it aggregate the
disk requirements of multiple concurrent flights.
Instead, it has a hardcoded minimum, which is enforced (by expiry) at
the start of each flight. It must be large enough for all in-progress
flights. This turns out not to be true any more, because of expansion
of both the number of different osstest branches, and because of
expansion of the number of jobs per flight, and because of expansion
of the activities within each job.
So, well, let's double it.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 1 Dec 2017 11:11:38 +0000 (11:11 +0000)]
cs-adjust-flight: Correct pseudo-BNF synopsys for runvar-build-set
In 497b2c6c933d13a05b01c6a654ce470be16dd78a
cs-adjust-flight: Rework runvar-build-set new value handling
the interpretation of this parameter was changed completely, but the
synopsis was not updated and thus became wrong.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Wed, 15 Nov 2017 11:00:07 +0000 (11:00 +0000)]
ts-xen-build-prep: Install libelf-dev for benefit of linux.git
Linux upstream has started needing libelf-dev. Without it, recent tip
fails (in our configuration) like this:
Makefile:938: *** "Cannot generate ORC metadata for CONFIG_UNWINDER_ORC=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel". Stop.
This is because kernel now is using objtool to create unwind
information. This needs libelf to work. Advantage is that this
approach no longer depends on assembler sources being heavily
annotated with unwind hints.
CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Stefano Stabellini <sstabellini@kernel.org> CC: Boris Ostrovsky <boris.ostrovsky@oracle.com> CC: Juergen Gross <jgross@suse.com> CC: Paul Durrant <paul.durrant@citrix.com> CC: Wei Liu <wei.liu2@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Juergen Gross <jgross@suse.com>
Roger Pau Monne [Wed, 25 Oct 2017 10:30:24 +0000 (11:30 +0100)]
osstest: Add freebsd runvars to examine job in generic flights
Previous patches only added the FreeBSD runvars to the jobs in the
examine flight, but failed to also add them to the examine job in the
generic flight.
This is necessary because the examine jobs now contain a step which
uses freebsd - the memdisk append determination.
On installations without an anointed freebsd job, and with no
freebsd-related config settings, that step will simply and harmlessly
fail.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Hook the memdisk parameter detection and the saving of the host
properties into the examine jobs.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v2:
- Do not pass a host ident to ts-examine-hostprops-save.
- Use .- for ts-memdisk-try-append so that the rest of the job will
run even if this step fails.
Changes since v1:
- Run the memdisk test first (so that we don't leave the host in a
weird state).
- Pass a host to the examine-hostprops-save.
ts-examine-hostprops-save: introduce a script to save properties
This script turns the properties stored in the runvars using the
format hostprop/$ident/$prop=$val into host properties stored in the
database.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v12:
- Introduce a dry_run variable and print a message only once if not
saving the properties in the database.
Changes since v2:
- Call selecthost based on the idents passed in the putative
hostprops runvar.
- Fix commit message.
- Use '/' instead of '_' in the runvars.
- Do a dry run if flight blessing != real.
- Fix parentheses indentation.
Changes since v1:
- Select a host for setting the properties.
- Print a message before exiting if blessing != real.
- Skip properties that don't contain the selected host.
ts-memdisk-try-append: introduce a script to test memdisk options
The intended usage is to run this script against every host in order
to record the possible needed memdisk flags.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v12:
- Add hyphens to the boot test options.
Changes since v1:
- Get the arch of the job and exit with 0 if it's not supported.
- Pass the --recordappend argument to ts-memdisk-try-append.
ts-freebsd-host-install: add arguments to test memdisk append options
This is needed in order to figure out which memdisk options should be
used to boot the images on each specific box.
Note that when passed the --record-append argument upon success the
script stores the tentative host property in the runvars.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v12:
- Add hyphens between words in command line options.
- Die if --record-append is specified without --test-boot.
Changes since v2:
- Fix commit message.
Changes since v1:
- Provide a --recordappend argument to force the recording the
memdisk parameters.
- Exit gracefully if a bootonly test is attempted against a
non-supported architecture.
- Use NONE instead of an empty string when calling
setup_netboot_memdisk if nothing should be appended.
- Do not perform any arch test in ts-freebsd-host-install.
This is used to store tentative host properties in the runvars of a
job, with the expectation that at some point (ie: at the end of the
job) they will be turned into real properties stored in the database.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v2:
- Use the following runvar format to store the putative host props:
hostprop/$ident/$prop=$val.
Roger Pau Monne [Mon, 23 Oct 2017 11:41:16 +0000 (12:41 +0100)]
netboot_memdisk: allow each host to have different append values
Some hosts require "append raw" [0] when booting with memdisk, while
others don't. This is based on the hardware/BIOS, and needs to be set
on a per-host basis.
In order to do this, add a new "MemdiskAppend" host property and make
use of it in the setup_netboot_memdisk helper.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v1:
- Explicitly use NONE for no options set (instead of an empty string,
which is the default).
- Allow to manually pass append parameters.
Roger Pau Monne [Mon, 23 Oct 2017 09:57:00 +0000 (10:57 +0100)]
HostDB: introduce set_property
And provide a helper in TestSupport to use it. This allows osstest to
set host properties from test script themselves (instead of using
the mg-hosts clu).
Note that the setting of host properties is limited to flights with
intended blessing real, and it will fail for any other blessing.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v12:
- Use intended blessing in the error message.
Changes since v2:
- Die if attempting to modify a host prop with intended blessing !=
real.
Roger Pau Monne [Mon, 23 Oct 2017 13:13:30 +0000 (14:13 +0100)]
osstest: save/retrieve the last successfully tested FreeBSD build
And use it in order to install the hosts for the next FreeBSD flight.
Note that the anoints are done after the push, but there's a recovery
mechanism in case osstest manages to do a successful push, but the
artifacts are not anointed for some reason. Osstest will continue to
create flights until the revision of the anointed artifacts matches
the one pushed to the repository (even if that means running flights
with OLD_REVISION == NEW_REVISION).
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v7:
- Add space between $freebsd_branch and * in list-prepared.
Changes since v6:
- Use just the branch name for anoint (instead of freebsd-$branch).
- Force the creation of flights if the revision of the current
anoints don't match OLD_VERSION.
- Iterate over the list of current anoints instead of open-coding the
arches.
- Split the retrieve/anoint code blocks into it's own sections.
Roger Pau Monne [Mon, 23 Oct 2017 13:03:21 +0000 (14:03 +0100)]
osstest: add a FreeBSD host install script
The installation is performed using the bsdinstall tool, which is part
of the FreeBSD base system. The installer image is setup with the
osstest ssh keys and sshd enabled by default, which allows the test
harness to just ssh into the box, create the install config file and
launch the scripted install.
Currently the installation is done with ZFS only, in stripe mode, and
a single disk.
In order to support the FreeBSD installer the setup_netboot_memdisk
helper is used, that allows setting the pxe boot of a host using a
memdisk.
The install script either picks the binary images from the output of a
previous FreeBSD buildjob (yet to be introduced), or from the folder
pointed by freebsd_distpath. This folder should contain the installer
image (install.img) and the compressed install sets (kernel.txz,
base.txz) together with the MANIFEST file that contains the checksums.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v4:
- Remove get_sets_path routine and xopts variable.
- Call resource_shared_mark_ready when the host is ready to be used.
Changes since v3:
- Use sha256file instead of the sha256sum clu.
- Shift parameters passed to the lock and copy script, so that
parameter 0 is ignored.
- Copy to .tmp and then move the FreeBSD image.
- Use a single-line find rune to cleanup old images.
- Add some missing "set -e".
- Don't check the output of target_cmd_output_root against undefined,
that's never going to be true.
- Split the setup_netboot_memdisk helper code into a pre-patch.
Changes since v2:
- Use sysrc instead of echo to modify rc.conf.
Changes since v1:
- Use pkg bootstrap instead of pkg update.
- Use freebsdbuildjob instead of freebsd_buildjob.
- add "append raw" to the memdisk command line.
- Use with-lock-ex instead of flock.
- Use system_checked and an array to hold the locked command to
execute.
- Use an array to store the list of possible disk devices.
- Use target_putfilecontents_root_stash in order to upload the
install script to the target.
- Remove the usage of TftpFreeBSDBase.
- Use target_tftp_prefix in order to get the prefix for the tftp
target image path.
- Add a missing linewrap.
- Replace the split used to get the runvar paths with a regexp.
- Place the install sets at /root/osstest_sets instead of
/tmp/osstest_sets.
- Pick the sets/installer image from the freebsdbuildjob
path_freebsdist or from the runvar variable freebsd_distpath.
- Add a comment about which FreeBSD specific inputs the script
consumes.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
Changes since v4:
- Add a deployment note to the commit message.
Changes since v3:
- New in this version (split from install patch).
Ian Jackson [Mon, 23 Oct 2017 10:04:54 +0000 (11:04 +0100)]
anointments: Provide ./mg-anoint list-prepared
Deployment note: Requires Text::Glob, which on Debian is in
libtext-glob-perl (which I have installed in the Xen Project and
Citrix Cambridge primary VMs).
CC: Roger Pau Monne <roger.pau@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Mon, 23 Oct 2017 10:04:34 +0000 (11:04 +0100)]
anointments: Introduce anointment scheme, and mg-anoint script
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
[Roger: fix conflict in README.planner after rebase] Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Ian Jackson [Thu, 19 Oct 2017 10:13:37 +0000 (11:13 +0100)]
mg-repro-setup: --autoalloc-nofree option
This lets you run the automatic allocator within your own task, so you
get to keep whatever it found.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Wei Liu <wei.liu2@citrix.com>
---
v3: Remove erroneous need to specify the HOSTSPEC "none:"
v2: Some docs, and a warning message.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 17 Oct 2017 11:08:31 +0000 (12:08 +0100)]
MaxUmask: enforce a maximum umask value
On some operating systems, the default umask is not 002 as it should
be (for the sensible setup with personal groups).
If a user with an 022 or 077 umask invokes osstest in Executive mode,
they end up creating directories in $c{Logs} which are writeable only
by them, and that can stop the whole system because the service user
cannot expire them.
Prevent this from happening.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 13 Oct 2017 13:11:56 +0000 (14:11 +0100)]
HostDiskRoot: bump to 20G
Some of our Windows guests have more RAM now, and some of them have
big ISOs too. The guest memory ends up in /root as a save image, and
the ISO ends up there too.
Double the size of / to 20G. That will probably do for now and is
unlikely to break anything.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Reported-by: Wei Liu <wei.liu2@citrix.com> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Roger Pau Monne [Wed, 11 Oct 2017 09:05:00 +0000 (10:05 +0100)]
pvh: rename pvh tests to pvhv2
Due to the recent changes to the PVH tests, all of them are now
failing because the current Linux kernel used by osstest doesn't
support PVHv2, and osstest treats the failures as regressions because
previously the PVH tests where actually testing classic PV.
Rename the tests to 'pvhv2' in order to prevent osstest from
classifying the failures as regressions.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 9 Oct 2017 11:30:31 +0000 (12:30 +0100)]
examine: save xl info output
One reason we want this is that it contains a reasonably easy-to-parse
record of the host memory.
When we have collected this information for all hosts, as xl info
output, we can write a program to copy the information into a host
property. This will allow us to restrict certain jobs to hosts with
enough memory.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: George Dunlap <George.Dunlap@citrix.com> CC: Roger Pau Monné <roger.pau@citrix.com>
Roger Pau Monne [Tue, 3 Oct 2017 10:21:35 +0000 (11:21 +0100)]
osstest: fix PVH DomU tests
The pvh=1 xl option was removed, so switch the PVH tests to use
type='pvh' instead.
On older branches still supporting PVHv1 (and the 'pvh' xl option),
this change would turn the PVH tests into PV tests), except that the
previous change removed the PVH tests from those branches.
PVHv1 has already been removed from upstream Linux and Xen.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Roger Pau Monne [Tue, 3 Oct 2017 10:21:37 +0000 (11:21 +0100)]
osstest: limit PVH tests to Xen >= 4.10
Older Xen versions don't support the 'type' xl option used to create a
PVHv2 guest, which we are about to switch to.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 1 Sep 2017 15:42:24 +0000 (16:42 +0100)]
PERLLIB, @INC: Use BEGIN { }
unshifting @INC is only effective if done at compile time, so that it
affects `use' directives. So "Perl: Put . on @INC" f261b07cbe96844dba5d8594c38149c0b179068a is simply broken
This patch was autogenerated using the following rune:
Ian Jackson [Fri, 1 Sep 2017 15:21:23 +0000 (16:21 +0100)]
mg-host: Prepare $checkhostq only if needed
This is helpful because in standalone mode there is no resources table
and this fails. But we want people to use `mg-hosts power' in
standalone mode, if they can, rather than running ./msu-pdu ad-hoc (or
whatever).
Reported-by: George Dunlap <George.Dunlap@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Fri, 1 Sep 2017 10:05:55 +0000 (11:05 +0100)]
production-config: Publish per-flight osstest git refs in separate tree
Those refs need to be somewhere public so that the relevant git
objects are accessible. I think they probably have to be in
refs/tags/ or refs/heads/ of some repo.
But it doesn't have to be the main one on xenbits. Most people won't
want these refs. Most git commits they are interested in (eg git
commits referred to in flight reports) will be available on the
osstest master branch. And having these refs there makes git fetch
and git clone of the main repo slow.
Deployment note:
I have already created the destination repo. Once this commit makes
it to master, I will manually transfer all the flight-* refs from the
main tree to osstest/osstest-massachusetts.git.
Reported-by: George Dunlap <George.Dunlap@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Tue, 1 Aug 2017 13:18:29 +0000 (14:18 +0100)]
mg-repro-setup: Slightly better document the alloc: syntax
Provide a clearer indication that the <resource-spec> is passed to
mg-allocate (and therefore, implicitly, that mg-allocate's docs should
be consulted).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Wei Liu <wei.liu2@citrix.com>
Ian Jackson [Fri, 28 Jul 2017 10:31:03 +0000 (11:31 +0100)]
arm64: tolerate host allocation failures
We have no working ARM64 hardware right now. Stop blocking things.
(Arguably this ought to be done in general for all arches, but
actually bugs in the flight construction code can cause this too, and
we wouldn't want such a regression to pass the osstest
self-push-gate.)
CC: Julien Grall <julien.grall@arm.com> CC: Stefano Stabellini <sstabellini@kernel.org> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Match @@-arm64 jobs (eg build-arm64) too
Ian Jackson [Wed, 26 Jul 2017 13:31:52 +0000 (14:31 +0100)]
coverity: Set CoverityToolsStripComponents to 2
This needs to be 2 for cov-analysis-linux64-2017.07.tar.gz
because that contains "./" at the start of its paths, which
tar counts as a component it's stripping.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Reviewed-by: Wei Liu <wei.liu2@citrix.com>
Ian Jackson [Mon, 22 May 2017 15:09:23 +0000 (16:09 +0100)]
ap-common: Switch to Linux 4.9 by default
I ran a special report[1] to see what to expect and:
Tests which did not succeed and are blocking,
including tests which could not be run:
test-amd64-i386-xl-qemuu-win7-amd64 15 guest-localmigrate/x10 fail REGR.
test-amd64-i386-xl-qemut-win7-amd64 15 guest-localmigrate/x10 fail REGR.
These Windows 7 migration tests have been failing on many branches and
don't look like they are something to do with the version of Linux
used in dom0.
Accordingly I intend to push this change to switch osstest to using
Linux 4.9 by default. ARM tests are not affected at this time.
CC: Boris Ostrovsky <boris.ostrovsky@oracle.com> CC: Juergen Gross <jgross@suse.com> CC: Stefano Stabellini <sstabellini@kernel.org> CC: Wei Liu <wei.liu2@citrix.com> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Roger Pau Monné <roger.pau@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Jun 2017 12:10:38 +0000 (13:10 +0100)]
step logfiles: Make substep_* take $logfile, not $script
The $script parameter was used by Standalone to print a message, but
this is no longer done.
It is used by Executive as the value for the "step" column in the
steps table, which shows up as "script" in the output from
sg-report-flight. But, when running under sg-run-job,
JobDB::Executive already has a better source of the script name: the
parent step (as reported in OSSTEST_TESTID). When not running under
sg-run-job the use of $0 is good enough.
Finally, it is used by Executive as a logfile name. But when running
under sg-run-job, JobDB::Executive gets the right logfile name anyway
(as above); and when not running under sg-run-job there is no default
logfile.
Conversely, substeps might organise their own logfile.
So abolish $script from the TestSupport substep API. Replace it in
substep_start and substep_eval with a $logfile parameter.
Update all call sites to no longer pass $script. None make their own
logfiles, so none pass the new parameter $logfile.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Jun 2017 12:05:18 +0000 (13:05 +0100)]
step logfiles: Honour new logfile column
There are the following places in the tree where we construct the
logfile name from stepno and step:
* tcl/JobDB-Executive main decision about logfile name } OK
* JobDB::Executive final fallback for new steps } keep
* steps-logfile-constraint.sql default for old data }
* sg-report-flight (twice) } both read stepno and step from the db;
* determine-failure-reasons } here we teach them to try logfile first
Now substep logs will have correct links in the job output, as soon as
the logfile column has the right data.
This is part of a "Populate-then-rely" schema change, as described in
schema/README.updates. Specifically, this is the step:
6. Optionally commit: code which reads new column, but which tolerates
it containing NULL/DEFAULT. (`add' is already Needed.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Jun 2017 12:02:24 +0000 (13:02 +0100)]
substep logfiles: Set logfile and step columns correctly for substeps
If we are running under sg-run-job, OSSTEST_TESTID will be set to the
testid of the parent step (ie, the step created by sg-run-job which
relates to the execution of the whole script).
Unless the caller has specified otherwise, the log and script are
the same as for the parent step.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Jun 2017 11:55:54 +0000 (12:55 +0100)]
substep logfiles: Populate new logfile column, at all, for substeps
For now we populate it with the sometimes-wrong value based on
$script. In detail:
* Add a parameter $logfile to $mjobdb->step_start, which currently
always gets passed undef.
* In JobDB::Executive::step_start, try to use it as the value for the
new logfile column. But, if it is not set (which is currently
always the case), default it to "<stepno>.<script>.log".
This is the same log filename currently assumed (and therefore used)
by sg-report-flight. So with current versions of sg-report-flight,
this has no visible functional change.
This is part of a "Populate-then-rely" schema change, as described in
schema/README.updates. Specifically, this is part of the step:
5. Commit: code to populate new column; changing `add' to status
Needed and `constraint' to status Ready.
We don't set `constraint' to Ready yet because we haven't completed
the code to populate the new column.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Jun 2017 11:43:00 +0000 (12:43 +0100)]
step logfiles: Provide new column
We will write things into it, and use it, soon.
This is the start of a "Populate-then-rely" schema change, as
described in schema/README.updates. Specifically, the steps:
1. Commit: new schema update `add', status Preparatory
2. Commit: new schema update `constraint', status Unfinished
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Jun 2017 11:08:48 +0000 (12:08 +0100)]
step logfiles: Standalone mode: Do not print uninformative $script value
Right now, this is sometimes wrong; even if it were right, it's not
going to be interesting in the transcript since the actual script name
is obvious from context.
Now this function does not care if $script is undef.
While we're here, fix the prototype (which is mostly commentary here,
as it is not used for method calls).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Jun 2017 12:40:00 +0000 (13:40 +0100)]
db write transactions: Make all INSERTs name columns explicitly
This makes the code more robust against schema updates. Specifically,
if any columns were to be deleted, these runes would break. (Also if
any column was inserted other than after the existing columns,
although this is not something Postgres can currently do AIUI.)
Affected tables are:
flights_harness_touched
jobs
resources
runvars
steps
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Thu, 29 Jun 2017 11:41:50 +0000 (12:41 +0100)]
schema updates: mg-schema-test-database: Cope with applied updates
The schema_updates table contents are written by the call to
mg-schema-update. Trying to copy the contents from the source db
correctly fails with duplicate key errors. So don't do that.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
projects / people / liuw / osstest.git / log