]> xenbits.xensource.com Git - people/iwj/osstest.git/log
people/iwj/osstest.git
6 years agomg-allocate: Don't try allocating the same resource for two specs wip.repro-flight-builds.v2
Ian Jackson [Fri, 3 May 2019 13:38:53 +0000 (13:38 +0000)]
mg-allocate: Don't try allocating the same resource for two specs

This is obviously wrong.  In fact it does not work (we bomb out in the
allocation).

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch

6 years agomg-repro-setup: Detect options (-...) in wrong place
Ian Jackson [Fri, 3 May 2019 13:17:02 +0000 (13:17 +0000)]
mg-repro-setup: Detect options (-...) in wrong place

Without this they tend to be interpreted as HOSTSPECs leading to
lossage.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch

6 years agomg-repro-flight: Provide --rebuild to make variant build jobs too
Ian Jackson [Thu, 2 May 2019 15:59:16 +0000 (16:59 +0100)]
mg-repro-flight: Provide --rebuild to make variant build jobs too

This allows a single command to repro a particular job with a variety
of different source code.

The implementation technique is:
  - run the build job in a separate flight, so that it can run
    with a separate task which gives its host up after the build
  - do much of the heavy lifting of runvar fiddling etc. in
    a new helper routine in cs-adjust-flight

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Add a missing `continue' (without which everything goes quite wrong)

6 years agomg-repro-setup: Move flight creation up before task creation
Ian Jackson [Thu, 2 May 2019 16:24:34 +0000 (17:24 +0100)]
mg-repro-setup: Move flight creation up before task creation

No significant functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agomg-repro-setup: Allow arguments to badusage
Ian Jackson [Thu, 2 May 2019 13:49:31 +0000 (14:49 +0100)]
mg-repro-setup: Allow arguments to badusage

No functional change with existing call sites.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agomg-repro-setup: Move logging setup to later
Ian Jackson [Thu, 2 May 2019 13:46:13 +0000 (14:46 +0100)]
mg-repro-setup: Move logging setup to later

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agomg-repro-setup: Break out compute_adjusts
Ian Jackson [Thu, 2 May 2019 13:39:44 +0000 (14:39 +0100)]
mg-repro-setup: Break out compute_adjusts

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agomg-repro-setup: Improve a doc message slightly
Ian Jackson [Thu, 2 May 2019 13:07:38 +0000 (14:07 +0100)]
mg-repro-setup: Improve a doc message slightly

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agocs-adjust-flight: Use db_prepare and honour -D -D for sql dumps
Ian Jackson [Thu, 2 May 2019 17:33:23 +0000 (18:33 +0100)]
cs-adjust-flight: Use db_prepare and honour -D -D for sql dumps

This makes debugging it easier.  No functional change with zero or one
occurrences of -D.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agocs-adjust-flight: Break out copy_jobs_*
Ian Jackson [Thu, 2 May 2019 15:58:14 +0000 (16:58 +0100)]
cs-adjust-flight: Break out copy_jobs_*

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agomg-execute-flight: Save an mro in tmp/
Ian Jackson [Thu, 2 May 2019 16:32:23 +0000 (17:32 +0100)]
mg-execute-flight: Save an mro in tmp/

This may be useful for some things.  For example, it will be used in
just a moment.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDrop Xen 4.5 and earlier
Ian Jackson [Wed, 1 May 2019 10:43:08 +0000 (11:43 +0100)]
Drop Xen 4.5 and earlier

These releases are out of security support.  They are known not to
build on Debian stretch, which is what we are using, and we do not
intend to ever update them to fix that.

Xen 4.6 is also out of security support but we want osstest to be able
to continue to build it so that we can test 4.6->4.7 migration, for
the purposes of testing Xen 4.7, which is still supported right now.

So we have recently applied some build fixes to the 4.6 tree, and for
now we retain 4.6 in osstest so that build fixes applied to
staging-4.6 can propagate to stable-4.6.

CC: committers@xenproject.org
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoRevert "arm64: tolerate host allocation failures"
Ian Jackson [Wed, 1 May 2019 10:14:21 +0000 (11:14 +0100)]
Revert "arm64: tolerate host allocation failures"

We now have 4 working arm64 boxes and this is quite ample for our
current test matrix.  We have two kinds of box, too.  So, great.

This reverts commit 0705bed9e20ea4375e7ac9f8109ef955ebc6fe9c.

CC: Stefano Stabellini <sstabellini@kernel.org>
Acked-by: Julien Grall <julien.grall@arm.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-kernel-build: Move main program to bottom of script
Ian Jackson [Fri, 26 Apr 2019 16:26:40 +0000 (17:26 +0100)]
ts-kernel-build: Move main program to bottom of script

Having it in the middle makes it quite hard to find !

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agocross builds: Build armhf kernels on amd64 hosts
Ian Jackson [Fri, 26 Apr 2019 14:58:47 +0000 (15:58 +0100)]
cross builds: Build armhf kernels on amd64 hosts

Our armhf hosts are devboards and very slow, as well as scarce.  It
takes 17ks or so for a kernel build.  This will go *much* faster on
an amd64 box and we have lots of those too.

standalone-generate-dump-flight-runvars shows that the only change is
to change host_arch from armhf to amd64 in build-armhf-pvops jobs.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
Acked-by: Julien Grall <julien.grall@arm.com>
---
v2: Fix typo in commit message.

6 years agocross builds: mfi-common: Prepare for kernel cross building
Ian Jackson [Fri, 26 Apr 2019 14:22:55 +0000 (15:22 +0100)]
cross builds: mfi-common: Prepare for kernel cross building

Introduce job_create_build_crossable, which takes a target->host
architecture map in its arguments, and use it for build-kern,
passing an empty architecture map.

Overall functional change is only to add
  host_arch=$arch
to the kernel build jobs, which has no ultimate effect because it's
the same as the arch=$arch.  (Difference in flight construction
verified with standalone-generate-dump-flight-runvars.)

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agocross builds: mfi-common: Break out set_build_hostflags
Ian Jackson [Fri, 26 Apr 2019 14:22:18 +0000 (15:22 +0100)]
cross builds: mfi-common: Break out set_build_hostflags

No functional change.

Verified with standalone-generate-dump-flight-runvars.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agocross builds: ts-kernel-build: Support cross target armhf
Ian Jackson [Fri, 26 Apr 2019 14:00:27 +0000 (15:00 +0100)]
cross builds: ts-kernel-build: Support cross target armhf

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Julien Grall <julien.grall@arm.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
---
v2: Drop HOSTCC= setting

6 years agots-kernel-build: Introduce cmd()
Ian Jackson [Fri, 26 Apr 2019 13:53:11 +0000 (14:53 +0100)]
ts-kernel-build: Introduce cmd()

Right now this is a simple wrapper around target_cmd_build.

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoarch replumbing: ts-debian-di-install: Use $gho->{Arch}
Ian Jackson [Fri, 26 Apr 2019 13:44:04 +0000 (14:44 +0100)]
arch replumbing: ts-debian-di-install: Use $gho->{Arch}

This is just tidying up.  The only effect is that now these would
honour $r{all_guest_arch} as a fallback.  But right now,
$r{GUEST_arch} will always be set, and that is what ends up in
$gho->{Arch}.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoarch replumbing: make-flight: Fix $r{arch} comment
Ian Jackson [Fri, 26 Apr 2019 13:35:07 +0000 (14:35 +0100)]
arch replumbing: make-flight: Fix $r{arch} comment

This comment was lamenting the very problem we are fixing now.  It
would now be possible to test i386->amd64 tools migration, by writing
an appropriate test job with different src_host_arch and
dst_host_arch etc.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoarch replumbing: Replace many $r{arch} with $[g]ho->{Arch}
Ian Jackson [Fri, 26 Apr 2019 13:27:12 +0000 (14:27 +0100)]
arch replumbing: Replace many $r{arch} with $[g]ho->{Arch}

No functional change with existing flights.  But the effect is that
now, generally, ts-* scripts and the support code will honour
host_arch, if it is set, in preference to arch.

This patch contains only replacements of $r{arch} with $ho->{Arch} or
$gho->{Arch}.  In fact, perhaps surprisingly, there were no locations
where $gho was wanted rather than $ho (I have double checked this).

Exceptions, where we left $r{arch} alone, are:

 * make-flight: a comment, which we are about to deal with;

 * ts-kernel-build: we are going to support cross building and
    $r{arch} is going to be the architecture of the kernel we want
    rather than of the build host.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoarch replumbing: ts-memdisk-try-append: Remove unidiomatic " "
Ian Jackson [Fri, 26 Apr 2019 13:24:18 +0000 (14:24 +0100)]
arch replumbing: ts-memdisk-try-append: Remove unidiomatic " "

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoarch replumbing: ts-debian-di-install: Remove unidiomatic { }
Ian Jackson [Fri, 26 Apr 2019 13:21:06 +0000 (14:21 +0100)]
arch replumbing: ts-debian-di-install: Remove unidiomatic { }

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoarch replumbing: Provide $ho->{Arch} and $gho->{Arch}
Ian Jackson [Fri, 26 Apr 2019 13:17:25 +0000 (14:17 +0100)]
arch replumbing: Provide $ho->{Arch} and $gho->{Arch}

With existing flights these are $r{arch} and GUEST_arch.
Nothing uses these yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Correctly set $gho->{Arch} in selectguest, not $ho->{Arch}

6 years agoarch replumbing: ts-host-install: Move $kern_arch_info setting
Ian Jackson [Fri, 26 Apr 2019 13:10:21 +0000 (14:10 +0100)]
arch replumbing: ts-host-install: Move $kern_arch_info setting

We are going to want to do this after selecthost.

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoTestSupport: target_var: Use host_V for host variables
Ian Jackson [Fri, 26 Apr 2019 13:07:08 +0000 (14:07 +0100)]
TestSupport: target_var: Use host_V for host variables

Change `target_var' to set `IDENT_V' rather than just V.  For
compatibility with older flights and older flight construction,
look for plain V too when looking up the variable.

And, we now look at all_host_V before V.  This has no functional
change with existing flights, because existing flights only have
  all_host_suite
  all_host_di_version
  all_host_os
and we never set the corresponding V form of those variables.

So with existing flights the only functional change is a change to
synth runvars, to add HOST_ to the name.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoTestSupport: target_var: Refactor to allow for another host case
Ian Jackson [Fri, 26 Apr 2019 13:05:49 +0000 (14:05 +0100)]
TestSupport: target_var: Refactor to allow for another host case

Make an explicit list of the prefixes and a loop to walk over them.
No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agostarvation: Better logging/debugging output
Ian Jackson [Thu, 19 Jul 2018 14:49:50 +0000 (15:49 +0100)]
starvation: Better logging/debugging output

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agohostalloc_maxwait_max: Use starvation most_optimistic
Ian Jackson [Tue, 16 Apr 2019 14:31:15 +0000 (15:31 +0100)]
hostalloc_maxwait_max: Use starvation most_optimistic

Now that we have a more sophisticated estimate of our likely scheduled
time, use it here too.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agostarvation: Abandon jobs which are unreasonably delaying their flight
Ian Jackson [Tue, 16 Apr 2019 14:30:29 +0000 (15:30 +0100)]
starvation: Abandon jobs which are unreasonably delaying their flight

Sometimes, due to a shortage of available resources, a flight might be
delayed because a handful of jobs are waiting much longer than the
rest.  Add a heuristic which causes these jobs to be abandoned.

We consider ourselves starving if we are starving now, based on the
most optimistic start time seen in the last I.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agostarvation: Infrastructure for jobs which are delaying their flights
Ian Jackson [Wed, 18 Jul 2018 12:21:27 +0000 (13:21 +0100)]
starvation: Infrastructure for jobs which are delaying their flights

Provide hostalloc_starvation_* in Osstest::Executive, and a comment
saying what we are going to do.  And provide a demo utility which
prints the effect of some particular runvar value on a range of
situations.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agostarvation: Use "starved" for hostalloc_maxwait_max
Ian Jackson [Wed, 18 Jul 2018 12:20:39 +0000 (13:20 +0100)]
starvation: Use "starved" for hostalloc_maxwait_max

Previously this was "broken".

We mustn't just call `broken' inside attempt_allocation because that
runs in a db transaction.  Instead, we arrange that attempt_allocation
returns 2, which threads its way back out to the return value from
alloc_resources, and then call broken there.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agots-host-allocate-Executive: Break out $now and add a newline
Ian Jackson [Tue, 16 Apr 2019 13:55:17 +0000 (14:55 +0100)]
ts-host-allocate-Executive: Break out $now and add a newline

Prep work, no functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agostarvation: Propagate starved job status into dependent jobs
Ian Jackson [Fri, 20 Jul 2018 15:13:23 +0000 (16:13 +0100)]
starvation: Propagate starved job status into dependent jobs

If a build job is starved, then the same status

(No jobs are marked `starved' yet.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agotcl/JobDB-Executive: Do not squash "starved" status
Ian Jackson [Tue, 16 Apr 2019 10:51:16 +0000 (11:51 +0100)]
tcl/JobDB-Executive: Do not squash "starved" status

ts-hosts-allocate is going to set the job status to `starved'
sometimes, and then die.  `starved' needs to be added to the list of
job statuses that sg-run-job leaves alone.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoTestSupport: Make "broken" set the step state too
Ian Jackson [Tue, 16 Apr 2019 17:49:25 +0000 (18:49 +0100)]
TestSupport: Make "broken" set the step state too

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agostarvation: Permit step_finish to set the state `starved'
Ian Jackson [Tue, 16 Apr 2019 17:56:54 +0000 (18:56 +0100)]
starvation: Permit step_finish to set the state `starved'

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoJobDB::Executive: step_*: fix log messages to talk about "steps"
Ian Jackson [Thu, 18 Apr 2019 16:11:17 +0000 (17:11 +0100)]
JobDB::Executive: step_*: fix log messages to talk about "steps"

These functions are about to be sometimes called for non-substep
steps.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoTestSupport: Make "broken" print the actual job state
Ian Jackson [Wed, 18 Jul 2018 12:19:56 +0000 (13:19 +0100)]
TestSupport: Make "broken" print the actual job state

No functional change except to a message.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agostep handling: Preserve step states set by ts-* scripts
Ian Jackson [Tue, 16 Apr 2019 17:46:42 +0000 (18:46 +0100)]
step handling: Preserve step states set by ts-* scripts

sg-run-job would unconditionally set the step state to the value it
calculated, which would usually be `pass' or `fail' or
`broken' (according to the recipe).

Relax this interface somewhat to allow a test script to set the step
status itself: specifically, do not overwrite an existing status of
  aborted broken starved

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agostarvation: Teach sg-execute-flight about job state starved
Ian Jackson [Tue, 16 Apr 2019 17:45:44 +0000 (18:45 +0100)]
starvation: Teach sg-execute-flight about job state starved

It needs to not mind if a job reports as `starved', even if sg-run-job
exited non-zero.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agostarvation: Teach ms-flights-summary about job state starved
Ian Jackson [Tue, 16 Apr 2019 17:44:23 +0000 (18:44 +0100)]
starvation: Teach ms-flights-summary about job state starved

This just affects sorting etc. in the summary display.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agostarvation: Teach archaeologists about starved job state
Ian Jackson [Thu, 18 Apr 2019 16:18:53 +0000 (17:18 +0100)]
starvation: Teach archaeologists about starved job state

sg-report-flight is a bit awkward.  It thinks mostly about step
status, not job status.  So, when justifying, if we don't find a step,
and the job state is starved, we treat the step as starved.

If there are only starved steps, then we don't have evidence that this
is a regression, because the test wasn't run in the baseline.

If there are other steps we look at those instead.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agostarvation: Teach sg-report-flight about starved step state
Ian Jackson [Thu, 18 Apr 2019 12:17:50 +0000 (13:17 +0100)]
starvation: Teach sg-report-flight about starved step state

We are going to use this for situations where the resources to run the
test weren't available.  In general we are going to treat this as not
a regression.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoalloc_resources: Support special abandonment values
Ian Jackson [Fri, 20 Jul 2018 15:11:37 +0000 (16:11 +0100)]
alloc_resources: Support special abandonment values

This gives a way for the caller's $resourcecall to signal something
interesting, back to its main loop.  This is useful for calling
broken, for example: that can't be done within $resourcecall because
$resourcecall operates within the allocation db transaction (which
ought to be rolled back...)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agots-logs-capture: Do not try to capture logs of hosts not allocated
Ian Jackson [Tue, 16 Apr 2019 10:50:41 +0000 (11:50 +0100)]
ts-logs-capture: Do not try to capture logs of hosts not allocated

Little significant change with existing non-broken flights etc.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoselecthost: Honour new $none_ok optional parameter
Ian Jackson [Tue, 16 Apr 2019 10:50:11 +0000 (11:50 +0100)]
selecthost: Honour new $none_ok optional parameter

No functional change with existing callers.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-hosts-allocate-Executive: with -U, just append to the same logfile
Ian Jackson [Fri, 20 Jul 2018 15:13:49 +0000 (16:13 +0100)]
ts-hosts-allocate-Executive: with -U, just append to the same logfile

This is much more convenient for ad-hoc use.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
6 years agoTftpDiVersion_stretch: update
Ian Jackson [Mon, 29 Apr 2019 10:36:51 +0000 (10:36 +0000)]
TftpDiVersion_stretch: update

We need to keep up with a d-i update by Debian.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agobuilds: Run i386 builds on amd64 kernels
Ian Jackson [Thu, 25 Apr 2019 15:04:56 +0000 (16:04 +0100)]
builds: Run i386 builds on amd64 kernels

Most hardware that supports i386 supports amd64 too.  When doing
builds we do need the right userland, but we don't actually care what
the kernel is doing.  With Linux 32-on-64 is good for that.

Especially, there is a kernel regression (evident in the Debian
stretch kernel, but not present in jessie's) where 32-bit Linux
mismanages the memory on hosts with moderately large amounts of
RAM (ie, significantly more RAM than can be addressed at once),
resulting in what amounts to a near-stall of the paging system.  Since
the paging system is used for filesystem writes too, the effect is
that commands run by builds can take totally unreasonable amounts of
time.  Ie, this version of Linux is broken when i386 PAE is needed.

In practice this is causing significant trouble in the Xen Project CI.

This kernel bug probably won't affect our test jobs because
(i) we use our own kernels, so we would probably detect this
regression when switching kernel branches etc. (ii) test jobs
run with a dom0_mem setting which avoids the preconditions for the
particular bug.

CC: Juergen Gross <jgross@suse.com>
CC: Boris Ostrovsky <boris.ostrovsky@oracle.com>
CC: Wei Liu <wei.liu2@citrix.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Paul Durrant <paul.durrant@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: preferred arch: Apply setarch to sshd
Ian Jackson [Thu, 25 Apr 2019 14:58:00 +0000 (15:58 +0100)]
Debian: preferred arch: Apply setarch to sshd

Many build systems (including Xen's, and autoconf) use uname to try to
discern the system's architecture.  When running i386 userland on an
amd64 kernel, this gives the wrong answer.  These build systems then
go off and try to do a sort of cross compile thing, and, generally,
fall over.

The uname -m value (which is what is at issue) is an inherited process
property.  Linux provides a utility `setarch' which changes this.  We
need to apply this to all builds; and it is not really convenient to
add an adverbial command to every build via the existing ssh build
shell rune mechanisms.

A fairly simple way to get the right behaviour is to wrap sshd
instead.  sshd doesn't mind what `personality' it sees.  Replacing
/usr/bin/sshd with a wrapper shell script might break
start-stop-daemon's attempts to shut down or restart sshd but we don't
care about that in osstest (certainly not on build installs, where
this feature is to be used).

Nothing uses this yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: preferred arch: Honour LinuxKernPreferredArchInfo
Ian Jackson [Thu, 25 Apr 2019 14:55:39 +0000 (15:55 +0100)]
Debian: preferred arch: Honour LinuxKernPreferredArchInfo

Nothing sets this yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agomg-repro-setup: Fix runvar adjustment shell syntax
Ian Jackson [Thu, 25 Apr 2019 14:56:11 +0000 (15:56 +0100)]
mg-repro-setup: Fix runvar adjustment shell syntax

-r would fail with a bash syntax error.  This bug has existed ever
since this feature was introduced.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agosg-run-job, ts-host-install: New --build option
Ian Jackson [Tue, 23 Apr 2019 10:49:24 +0000 (11:49 +0100)]
sg-run-job, ts-host-install: New --build option

Used to specify that the host will be used for building.  Currently
has no effect.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-syslog-server: --no-stdin option
Ian Jackson [Thu, 18 Apr 2019 14:52:59 +0000 (15:52 +0100)]
ts-syslog-server: --no-stdin option

Useful when running on a tty interactively.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agocr-daily-branch: ovmf: "usually" use xen-tested-master
Ian Jackson [Thu, 18 Apr 2019 11:18:10 +0000 (12:18 +0100)]
cr-daily-branch: ovmf: "usually" use xen-tested-master

This branch is supposed to be suitable for all versions of Xen.
Conversely, older versions of OVMF do not build on newer compilers (as
provided, eg, in stretch).

So, for "branches" other than xen-unstable and xen-unstable-smoke, use
the usual "determine_version" machinery, which will select
xen-tested-master for branches other than the ovmf branch itself.

No change for the ovmf "branch", nor for xen-unstable*.  The effect is
to switch xen-*-testing, qemu-*, linux-*, etc., to all use ovmf
xen-tested-master.

Verified with standalone-generate-dump-flight-runvars.  In detail:
In these flights:
   examine freebsd-master libvirt linux-*
   osstest qemu-* seabios xen-4.* xtf
in these jobs
   build-amd64
   build-arm64
   build-armhf
   build-i386
change
   revision_ovmf  to   the output of   ./ap-fetch-version-baseline ovmf
   tree_ovmf      to   git://xenbits.xen.org/osstest/ovmf.git
(each being previously set to the empty string).

Acked-by: Anthony PERARD <anthony.perard@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: Work around Debian stretch bug with disk device names
Ian Jackson [Wed, 17 Apr 2019 16:14:41 +0000 (17:14 +0100)]
Debian: Work around Debian stretch bug with disk device names

The Debian stretch i386 installer does not have Xen PV-on-HVM drivers.
When we run the HVM guest installer, it therefore sees /dev/sda (the
emulated IDE).  But the booted system *has* the drivers and sees
/dev/xvda.

The Debian installer is supposed to put UUID= in the bootloader config
and so on in the installed system.  However, this does not work due to
a bug (Debian #852323).  Work around it.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-debian-hvm-install: Disable net.ifnames for Debian HVM guests
Ian Jackson [Wed, 17 Apr 2019 14:10:30 +0000 (15:10 +0100)]
ts-debian-hvm-install: Disable net.ifnames for Debian HVM guests

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-debian-hvm-install: Use ---, and no longer duplicate $gconsole
Ian Jackson [Wed, 17 Apr 2019 14:05:32 +0000 (15:05 +0100)]
ts-debian-hvm-install: Use ---, and no longer duplicate $gconsole

In October 2014, debian-installer was updated to honour `---' in the
way it previously honoured `--': #762007.  If we use that instead,
we don't need to duplicate $gconsole any more.

This, effectively, reverts
  db26f5825a21269d9218417a9ca40bc5d47755d2
  ts-host-install: include console before *and* after the -- marker.

In principle this ought to be changed only in "recent enough" Debian
suites, but certainly at least jessie has the fix and jessie is quite
broken now, so let's not bother with making it conditional.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agomg-repro-setup: Allow, and advertise, ^ for unsetting runvars
Ian Jackson [Wed, 17 Apr 2019 11:11:51 +0000 (12:11 +0100)]
mg-repro-setup: Allow, and advertise, ^ for unsetting runvars

This avoids any possible problems with shells misinterpreting ! and is
consistent with the other recent changes replacing ! for negation
with ^.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agodebianhvm tests: Re-permute guest architecture
Ian Jackson [Tue, 16 Apr 2019 18:33:35 +0000 (19:33 +0100)]
debianhvm tests: Re-permute guest architecture

Firstly, change the smoke flight to use amd64, not i386.
Secondly, make the other flights have a mixture of i386 and amd64, not
just amd64.

It was always wrong to test only one guest architecture in the main
flights.

The context for changing the smoke flight is that there seems to be a
problem affecting Debian i386 guests.  This is rather a nuisance but
it is not a Xen regression so we probably want to do an amd64 guest
instead so we have some HVM tests in the smoke flights.

Checked with standalone-generate-dump-flight-runvars.  Differences
are as follows:

1. Generally, to change the guest arch from amd64 to i386 in:

  test-amd64-amd64-xl-qemut-debianhvm-{amd64,i386}-xsm
  test-amd64-amd64-xl-qemuu-debianhvm-{amd64,i386}-xsm
  test-amd64-i386-xl-qemut-debianhvm-{amd64,i386}-xsm
  test-amd64-i386-xl-qemuu-debianhvm-{amd64,i386}-xsm

This occurs in all the usual flights.

2. To change the guest arch from i386 to amd64 in:

  xen-unstable-smoke   test-amd64-amd64-xl-qemuu-debianhvm-i386

In all cases, this implies a variety of changes to runvars, including
the ISO to be used, the install timeout, etc.

Usually I would introduce new jobs first, and then, after an overlap,
remove old ones.  But the regression, due to the i386 hvm stretch
problem, is already with us.

I expect this change to unblock the xen-unstable-smoke flights by
dropping the failing job and replacing it with a passing one.

The other flights will now start to fail, though.  I will need,
separately, to develop a workaround for the stretch i386 hvm guest
bug.

CC: Wei Liu <wei.liu2@citrix.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Juergen Gross <jgross@suse.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-host-install: Do not force MAC address of Xen vifs
Ian Jackson [Mon, 15 Apr 2019 11:46:09 +0000 (12:46 +0100)]
ts-host-install: Do not force MAC address of Xen vifs

For some broken hosts we explicitly force the mac address in
software.  (This is controlled by the ForceMacAddress host property.)

We achieve this by writing a udev rule (which ends up both in the
installer and in the initramfs) which calls `ip link set ... address'.

However: in some cases we share the initramfs with guests.  In
particular, the `xen-tools' package likes to use the host initramfs.

In stretch at least, this rule, empirically, applies in the guest too.
The result is that with ForceMacAddress, the guest ends up with the
same mac address as the host, breaking everything.

Fix this by explicitly excluding Xen network interfaces.  These all
get a proper mac address from xenstore (ultimately, from osstest
itself via the guest config, or in other configurations from libxl).

I don't know why this apparently worked on jessie.  Startup order
changes are a possibility.

CC: Julien Grall <julien.grall@arm.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoSwitch to Debian Stretch
Wei Liu [Tue, 31 Oct 2017 09:22:57 +0000 (09:22 +0000)]
Switch to Debian Stretch

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoproduction-config-cambridge: Provide TftpDiVersion_stretch
Ian Jackson [Wed, 10 Apr 2019 10:52:21 +0000 (11:52 +0100)]
production-config-cambridge: Provide TftpDiVersion_stretch

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: Fix /lib/udev/ifupdown-hotplug in guests from debootstrap
Ian Jackson [Tue, 9 Apr 2019 16:52:58 +0000 (17:52 +0100)]
Debian: Fix /lib/udev/ifupdown-hotplug in guests from debootstrap

The ifupdown-hotplug problem just described affects guests too.

Empirically, this occurs only with the guests from xen-tools.  In my
osstest stretch series development tests this resulted in occasional
failures of ts-guest-start.  The problem is not deterministic; and it
may be that it is a race between the guest's random dhcp retry and
osstest's timeout, or something.  The race affected both x86 and ARM.

I conjecture that it only affects xen-tools created guests because
those guests use sysvinit.  Presumably the other guests have systemd
which has some different ordering.  I conjecture that the sysvinit
boot arrangements were damaged by some changes made in stretch to
shared components (udev, probably) to support systemd.

An alternative explanation for seeing the failure only with xen-tools
created guests might be that all guests are affected, but only
xen-tools created guests are booted with a short timeout; whereas the
d-i created ones have a long timeout for firstboot at least.  I don't
believe this theory because all guests are restarted with
ts-guest-start and a short timeout.

So: in ts-debian-fixup, execute the sed rune to fix
/lib/udev/ifupdown-hotplug.  This then happens before first boot.

Debian guests installed via d-i are not affected by this patch.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: Fix /lib/udev/ifupdown-hotplug to not run if / is ro
Ian Jackson [Tue, 9 Apr 2019 16:43:33 +0000 (17:43 +0100)]
Debian: Fix /lib/udev/ifupdown-hotplug to not run if / is ro

Empirically, without this, on Debian stretch:

 1. udev starts before / is remounted rw
 2. udev spots eth0 and runs /lib/udev/ifupdown-hotplug
 3. ifupdown-hotplug runs ifup which runs isc-dhcp-client
 4. isc-dhcp-client talks to the dhcp server to get an address
 5. isc-dhcp-client tries to write the lease and run the hook
    scripts but something here fails with EROFS
 6. isc-dhcp-client sends DHCPDECLINE to the server
 7. GOTO 4

This loop continues for several seconds, until / is remounted rw.

None of this is appears in any of the guest's logs because syslog is
not running yet, and none of this stuff goes to the console either.
But it does appear in the dhcp *server* logs so that a conscientious
administrator will suffer consternation and concern.

It is not ever sensible to run /lib/udev/ifupdown-hotplug with /
mounted ro.  (Maybe it is not sensible to run udev so early.)
Skipping it is fine because the boot sequence contains an explicit
call to ifup which occurs *after* / is remounted, and that will
collect any interfaces which were skipped earlier.

In my osstest stretch series development tests I don't think I saw any
actual host install failures due to this situation.  The timeouts are
generous enough not to matter, and of course when we install Xen we
reconfigure the host networking to have a static IP address so then
the problem goes away.

In this patch we do this for the host.  We provide a function to
return the appropriate rune which we will use in a moment.

I have not yet reported this situation to the appropriate Debian
channels.  That's on my backlog.  But in any case I have limited the
workaround to stretch so we will revisit this for buster.

The approach to fixing this is somewhat awkward.

Firstly, since the bug is in /lib/udev/ifupdown-hotplug we want to
edit that script.  But we need to do it in the installer environment
as a late_command, because after first boot, via ssh, is too late.
The installer environment has no `patch'.  I didn't want to just
supply a whole new script.  So instead we use sed and mv.

Secondly, as for the contents of /lib/udev/ifupdown-hotplug: I wasn't
able to think of a convenient shell command which will tell us the
errno value from trying to write a file.  Plenty will print the
strerror but I balked at LC_MESSAGES=C and grep.  Perl, however, can
do this, and is always available on Debian.  So perl it is.

Thirdly, the code has a bad case of toothpicks (lots of \), because it
needs to pass through (i) perl (ii) the shell (iii) sed's regexp
syntax and/or i command.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agodm restrict, fishdescriptor: Update to a fixed chiark-scripts
Ian Jackson [Mon, 8 Apr 2019 16:27:14 +0000 (17:27 +0100)]
dm restrict, fishdescriptor: Update to a fixed chiark-scripts

I have only just fixed a bug which stops our test from working
and the fix is not upstream yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoplatforms: Honour suite in get_arch_platforms
Ian Jackson [Mon, 8 Apr 2019 16:07:42 +0000 (17:07 +0100)]
platforms: Honour suite in get_arch_platforms

The available platforms may depend on the suite to be used.

Actually implement that for HostDB::Executive.
For Static, we leave it to the user.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoplatforms: Pass suite to get_arch_platforms
Ian Jackson [Mon, 8 Apr 2019 15:03:07 +0000 (16:03 +0100)]
platforms: Pass suite to get_arch_platforms

The available platforms may depend on the suite to be used.

For now we use $defsuite from make-flight, which is not entirely right
but it will do for now because we don't use other suites much.

No functional change yet since neither implementation uses it.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agomake-flight: shadow test: Disable kpti in guests
Ian Jackson [Fri, 5 Apr 2019 16:08:55 +0000 (17:08 +0100)]
make-flight: shadow test: Disable kpti in guests

Since Spectre/Meltdown, shadow has been a lot slower, especially with
KPTI in the guest.  Empirically, too slow (with the kernel from Debian
stretch).

CC: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-debian-hvm-install: Honour linux_boot_append target var
Ian Jackson [Fri, 5 Apr 2019 16:07:37 +0000 (17:07 +0100)]
ts-debian-hvm-install: Honour linux_boot_append target var

This looks for:
  <guest>_linux_boot_append
  all_guest_linux_boot_append

Nothing sets these yet.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-xen-install: Install libpciaccess0
Ian Jackson [Fri, 5 Apr 2019 15:33:11 +0000 (16:33 +0100)]
ts-xen-install: Install libpciaccess0

In "ts-xen-build-prep: install packages for suites >jessie",
libpciaccess-dev was added for the benefit of qemu.  libvirt needs it
too.

We also need the runtime library.  Without it, libvirt does not start:

  2019-04-04 22:35:36.760+0000: 3623: error : virModuleLoadFile:53 : internal error: Failed to load module '/usr/local/lib/libvirt/connection-driver/libvirt_driver_nodedev.so': libpciaccess.so.0: cannot open shared object file: No such file or directory

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agotarget_install_packages: Consistently use qw(...) rather than '...'
Ian Jackson [Fri, 5 Apr 2019 15:31:20 +0000 (16:31 +0100)]
target_install_packages: Consistently use qw(...) rather than '...'

qw(...) splits its argument into words.

There is one semantic change, where two package names were passed in a
single argument.  That worked by accident.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agodm restrict audit: always install (some) chiark-scripts
Ian Jackson [Fri, 5 Apr 2019 13:57:54 +0000 (14:57 +0100)]
dm restrict audit: always install (some) chiark-scripts

In
  dm restrict audit: install newer chiark-scripts for fishdescriptor
arrangements were made to install suitable chiark-scripts for
for jessie and stretch.

For buster and later, the mainline Debian version of chiark-scripts is
indeed sufficient, but nothing installed it.  Do that.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agodm restrict audit: actually install fishdescriptor in host
Ian Jackson [Fri, 5 Apr 2019 13:55:53 +0000 (14:55 +0100)]
dm restrict audit: actually install fishdescriptor in host

In
  dm restrict audit: install newer chiark-scripts for fishdescriptor
arrangements were made to install a backport of chiark-scripts
but the code was mistakenly placed in preseed_create_guest but
of course it's needed in the host.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: Move preseed_backports_packages earlier
Ian Jackson [Fri, 5 Apr 2019 13:55:32 +0000 (14:55 +0100)]
Debian: Move preseed_backports_packages earlier

No functional change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agodm restrict audit: actually install right package for fishdescriptor
Ian Jackson [Fri, 5 Apr 2019 13:04:26 +0000 (14:04 +0100)]
dm restrict audit: actually install right package for fishdescriptor

In
  dm restrict audit: install newer chiark-scripts for fishdescriptor
a locally-provided chiark-scripts_6.0.2_all.deb was installed for
jessie.  For stretch a backport was installed, but mistakenly
of chiark-utils-bin rather than chiark-scripts.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: Add reference to bug numbers for erase-other-disks
Ian Jackson [Fri, 5 Apr 2019 12:58:21 +0000 (13:58 +0100)]
Debian: Add reference to bug numbers for erase-other-disks

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: set partman-lvm/device_remove_lvm_span
Ian Jackson [Fri, 5 Apr 2019 12:31:07 +0000 (13:31 +0100)]
Debian: set partman-lvm/device_remove_lvm_span

Web searching[1] suggests that this suppresses this error:

  !! ERROR: Unable to automatically remove LVM data
  Because the volume group(s) on the selected device also consist of
  physical volumes on other devices, it is not considered safe to
  remove its LVM data automatically. If you wish to use this device
  for partitioning, please remove its LVM data first.

[1] eg https://serverfault.com/questions/571363/unable-to-automatically-remove-lvm-data

It doesn't, though.  I am only experiencing it now because the ad-hoc
disk-erasing (25erase-other-disks) is broken for other reasons.  But
let's have it anyway as it sounds like a thing we might want.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: partman scripts: Run right away too
Ian Jackson [Fri, 5 Apr 2019 12:24:52 +0000 (13:24 +0100)]
Debian: partman scripts: Run right away too

We are switching the installation of these to partman/early_command
which runs as a result of a /lib/partman/init.d hook.  That means that
things we install don't get picked up, so run them right away (too).

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoDebian: preseed_hook_installscript: New $atonce option
Ian Jackson [Fri, 5 Apr 2019 12:24:01 +0000 (13:24 +0100)]
Debian: preseed_hook_installscript: New $atonce option

This runs the script right away as well as merely installing it.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agopreseed_hook_installscript: Use partman/early_command, not preseed/
Ian Jackson [Thu, 4 Apr 2019 16:32:39 +0000 (17:32 +0100)]
preseed_hook_installscript: Use partman/early_command, not preseed/

On iso-based installs, with stretch, preseed/early_command runs before
the network is up.  This causes the install to fail.

Our existing call sites add things to
   /usr/lib/base-installer.d/
   /lib/partman/init.d/
for which this is still early enough.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agopreseed_hook_command: allow specifying di keys other than preseed/*
Ian Jackson [Thu, 4 Apr 2019 16:30:08 +0000 (17:30 +0100)]
preseed_hook_command: allow specifying di keys other than preseed/*

Ie, only add preseed/ if there is not already a slash.

No functional change with existing call sites other than urls and
temporary filenames.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-debian-fixup: Use debian_overlays_fixup_cmd
Ian Jackson [Wed, 3 Apr 2019 15:44:38 +0000 (16:44 +0100)]
ts-debian-fixup: Use debian_overlays_fixup_cmd

Otherwise we get the same error for guests as was prevented for hosts
by "preseed_base: chmod ssh host private keys to placate sshd".

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agopreseed_base: break out debian_overlays_fixup_cmd
Ian Jackson [Wed, 3 Apr 2019 15:42:55 +0000 (16:42 +0100)]
preseed_base: break out debian_overlays_fixup_cmd

We are going to want this for guests too.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agopreseed_base: chmod ssh host private keys to placate sshd
Ian Jackson [Tue, 2 Apr 2019 15:24:08 +0000 (16:24 +0100)]
preseed_base: chmod ssh host private keys to placate sshd

Otherwise:
  Could not load host key: /etc/ssh/ssh_host_ecdsa_key
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
  It is recommended that your private key files are NOT accessible by others.
  This private key will be ignored.
  key_load_private: bad permissions
  Could not load host key: /etc/ssh/ssh_host_ed25519_key

This seems to start happening with stretch.  Presumably stretch is
more annoyingly picky than jessie.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-guests-nbd-mirror: Use target_getfile_root_stash
Ian Jackson [Fri, 30 Nov 2018 16:45:21 +0000 (16:45 +0000)]
ts-guests-nbd-mirror: Use target_getfile_root_stash

That removes the rather ad-hoc open-coding.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoTestSupport: Provide target_getfile_root_stash
Ian Jackson [Fri, 30 Nov 2018 16:44:36 +0000 (16:44 +0000)]
TestSupport: Provide target_getfile_root_stash

This convenient function selects a local filename based on a target
filename, and copies the target file to the selected stash file.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoTestSupport: target_somefile_leaf rename and change a variable
Ian Jackson [Fri, 30 Nov 2018 16:42:26 +0000 (16:42 +0000)]
TestSupport: target_somefile_leaf rename and change a variable

Rename this function.  `getleaf' contains `get' which makes it sound
like the function copies something, or returns answers suitable for
getting, or something.

Also rename `$rdest' to `$rfile' since it might be a source too.
(Although we are not about to make it a source...)

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoTestSupport: Move `target_somefile_getleaf' earlier
Ian Jackson [Fri, 30 Nov 2018 16:41:14 +0000 (16:41 +0000)]
TestSupport: Move `target_somefile_getleaf' earlier

We are going to make more use of this in intervening code.

Pure code motion.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agoTestSupport: Move `stashed files' and `next_unique_name' earlier
Ian Jackson [Fri, 30 Nov 2018 16:39:13 +0000 (16:39 +0000)]
TestSupport: Move `stashed files' and `next_unique_name' earlier

We are going to make more use of this in intervening code.

Pure code motion.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-xen-build: Enable ITS driver in Xen
Julien Grall [Fri, 30 Nov 2018 15:23:42 +0000 (15:23 +0000)]
ts-xen-build: Enable ITS driver in Xen

The ITS driver was added in Xen 4.10 as a technical preview feature.
However, it is required in order to boot Xen as Thunder-X because
PCI devices don't support legacy interrupt.

So enable CONFIG_ITS in our Xen build.

Signed-off-by: Julien Grall <julien.grall@arm.com>
6 years agots-kernel-build: Enable some additional drivers for Thunder-X
Ian Jackson [Fri, 30 Nov 2018 15:18:32 +0000 (15:18 +0000)]
ts-kernel-build: Enable some additional drivers for Thunder-X

Without this, our kernels do not find the storage.

Suggested-by: Julien Grall <julien.grall@arm.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agopersistent-net: Set net.ifnames=0 in di_installcmdline_core
Ian Jackson [Wed, 21 Nov 2018 16:14:13 +0000 (16:14 +0000)]
persistent-net: Set net.ifnames=0 in di_installcmdline_core

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
6 years agopersistent-net: Include initramfs script to copy to target
Ian Jackson [Thu, 29 Nov 2018 16:00:56 +0000 (16:00 +0000)]
persistent-net: Include initramfs script to copy to target

This is the piece which actually copies the installer's network names
to the target.  It should not appear on the installed system, so it's
not in overlay-persistent-net.

Technically this is only useful when the installer has the
overlay-persistent-net in it, which is done only in ts-host-install
and not in all the places where setup_netboot_firstboot is used.
But without overlay-persistent-net it is harmless, and it is most
convenient to put it here.

The little script fragment was copied out of a jessie debian-installer
initramfs environment.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-host-install: Put canary in 70-persistent-net.rules
Ian Jackson [Thu, 29 Nov 2018 18:01:26 +0000 (18:01 +0000)]
ts-host-install: Put canary in 70-persistent-net.rules

This will allow us to see if the initramfs's network names are being
properly copied to the installed system.  Ie, this is just a debugging
aid.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-host-install: Unconditionally mkdir -p /etc/udev/rules.d
Ian Jackson [Tue, 2 Apr 2019 14:50:29 +0000 (15:50 +0100)]
ts-host-install: Unconditionally mkdir -p /etc/udev/rules.d

We are going to want this directory to exist so that we can put a
canary in 70-persistent-net.rules.

In the cases where the behaviour of osstest changes, the empty
directory does not result in any overall change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agots-host-install: Break out $persistent_net_rules
Ian Jackson [Thu, 29 Nov 2018 18:00:55 +0000 (18:00 +0000)]
ts-host-install: Break out $persistent_net_rules

We're going to want to reuse this value.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
6 years agopersistent-net: Add overlay in installer >= stretch
Ian Jackson [Tue, 27 Nov 2018 14:33:41 +0000 (14:33 +0000)]
persistent-net: Add overlay in installer >= stretch

We are going to need this in the installer so that the interface names
from the installer environment are captured so that they can be the
same on the host.

This prepares the ground for turning off net.ifnames.  The actual
rules are gated on net.ifnames so right now there is no change.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>