]> xenbits.xensource.com Git - people/liuw/osstest.git/log
people/liuw/osstest.git
9 years agoSchema: Support database schema updates
Ian Jackson [Thu, 10 Dec 2015 13:26:00 +0000 (13:26 +0000)]
Schema: Support database schema updates

See schema/README.schema, introduced in this patch, for the design.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Add comment about test db safety catch.

v3: Fix spurious message from ./mg-schema-updates apply.
    Fix grammar error in README.updates.

v2: Slight increase schema update name length format.
    Docs fixes:
    Change erroneous `three' to `four'.
    Change `state' to `status' throghout.
    Explain scope of <status>.
    Sort out (and renumber) `Update order for Populate-then-rely'.
    Sort out "Statuses" explanations.
    Encourage use of DML update, rather than ad-hoc scripts,
     for populating new columns.

9 years agoSchema: Introduce mg-schema-create
Ian Jackson [Thu, 10 Dec 2015 12:29:32 +0000 (12:29 +0000)]
Schema: Introduce mg-schema-create

There is a fair amount of option parsing clobber here that will be
relevant shortly.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoSchema: Remove SET OWNER and GRANT/REVOKE from schema/initial.sql
Ian Jackson [Thu, 10 Dec 2015 12:13:58 +0000 (12:13 +0000)]
Schema: Remove SET OWNER and GRANT/REVOKE from schema/initial.sql

Really, we don't want the initial schema setup to mess about with
permissions.  Instead, we simply expect to run the creation as the
correct role user.

So:
 - Remove the code in mg-schema-test-database to remove the
   permission settings from initial.sql;
 - Instead, run exactly that code on initial.sql and commit the
   result.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoSchema: Rename schema file
Ian Jackson [Mon, 7 Dec 2015 18:25:14 +0000 (18:25 +0000)]
Schema: Rename schema file

We are going to have multiple schema snippets and this is going be
just the initial baseline.

Rename the file and change references to it.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agomg-schema-test-database: Fix argument parsing for _SUFFIX
Ian Jackson [Wed, 9 Dec 2015 12:04:03 +0000 (12:04 +0000)]
mg-schema-test-database: Fix argument parsing for _SUFFIX

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoExecutive DB: Reduce strength of DB locks
Ian Jackson [Fri, 11 Dec 2015 16:13:00 +0000 (16:13 +0000)]
Executive DB: Reduce strength of DB locks

The purpose of these locks is partly to prevent transactions being
aborted (which I'm not sure the existing code would in practice cope
with, although this is a bug) and also to avoid bugs due to the fact
that
  SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
does not mean that the transactions are necessarily serialisable!
  http://www.postgresql.org/docs/8.3/static/transaction-iso.html

In SQL in general it is possible for read-only transactions to
conflict with writing transactions.

However, in PostgreSQL this is not a problem because Postgres uses
multi-version concurrency control: it retains the old version of the
data while the read transaction is open:
  http://www.postgresql.org/docs/8.3/static/transaction-iso.html

So a read transaction cannot cause a write transaction to abort, nor
vice versa.  So there is no need to have the database explicit table
locks prevent concurrent read access.

Preventing concurrent read access means that simple and urgent updates
can be unnecessarily delayed by long-running reader transactions in
the history reporters and archaeologists.

So, reduce the lock mode from ACCESS EXCLUSIVE to ACCESS.  This still
conflicts with all kinds of updates and prospective updates, but no
longer with SELECT:
  http://www.postgresql.org/docs/8.3/static/explicit-locking.html

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix grammar and typo in commit message.

9 years agoExecutive DB: Eliminate SQL locking for read-only transactions
Ian Jackson [Fri, 11 Dec 2015 16:04:11 +0000 (16:04 +0000)]
Executive DB: Eliminate SQL locking for read-only transactions

Our transactions generally run with
  SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
(which, incidentally, does not mean that the transactions are
necessarily serialisable!)

In SQL in general it is possible for a read-only transaction to fail
and need to be retried because some writer has updated things.

However, in PostgreSQL this is not possible because Postgres uses
multi-version concurrency control: it retains the old version of the
data while the read transaction is open:
  http://www.postgresql.org/docs/8.3/static/transaction-iso.html

(And, of course, SQLite uses MVCC too, and all transactions in SQLite
are fully serialisable.)

So it is not necessary for these read-only operations to take out
locks.  When they do so they can unnecessarily block other important
work for long periods of time.

With this change, we go further from the ability to support databases
other than PostgreSQL and SQLite.  However, such support was very
distant anyway because of differences in SQL syntax and semantics, our
reliance in Executive mode on Postgres's command line utilities, and
so on.

We retain the db_retry framing because (a) although the retry loop is
not necessary in these cases, the transaction framing is (b) it will
make it slightly easier to reverse this decision in the future if we
ever decide to do so (c) it is less code churn.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix minor error in in commit message

9 years agoExecutive: Permit OSSTEST_TASK=<refkey> (for static tasks)
Ian Jackson [Tue, 8 Dec 2015 14:11:47 +0000 (14:11 +0000)]
Executive: Permit OSSTEST_TASK=<refkey> (for static tasks)

If OSSTEST_TASK is not set, we construct a <refkey> from the username
and the nodename, and look for a such a static task.  If OSSTEST_TASK
/is/ set would require it to contain `<taskid> <type> <refkey>'.

In this patch, permit OSSTEST_TASK to be set simply to the <refkey>.
This is much more convenient and doesn't involve manually looking up
taskids.  The risk of error seems negligible.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agomg-execute-flight: create tmp directory before using it
Wei Liu [Mon, 7 Dec 2015 14:51:07 +0000 (14:51 +0000)]
mg-execute-flight: create tmp directory before using it

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomg-allocate: fix comment for deallocation
Wei Liu [Mon, 7 Dec 2015 14:38:56 +0000 (14:38 +0000)]
mg-allocate: fix comment for deallocation

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomg-hosts: document "power" command
Wei Liu [Mon, 7 Dec 2015 17:24:23 +0000 (17:24 +0000)]
mg-hosts: document "power" command

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agostandalone: Log things we are running via with_logging
Ian Campbell [Fri, 4 Dec 2015 15:27:59 +0000 (15:27 +0000)]
standalone: Log things we are running via with_logging

Turning on set -x generally in this script is too verbose, so run the
command in a subshell which sets -x.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomg-schema-test-database: Add workflow doc comment
Ian Jackson [Mon, 7 Dec 2015 17:06:12 +0000 (17:06 +0000)]
mg-schema-test-database: Add workflow doc comment

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch

9 years agomg-schema-test-database: Safety catch in JobDB database open
Ian Jackson [Mon, 7 Dec 2015 16:33:57 +0000 (16:33 +0000)]
mg-schema-test-database: Safety catch in JobDB database open

When we open the `osstest' database, see whether this is a parent DB
(main DB) from which a test DB has been spawned by this user.

If it has, bomb out, unless the user has specified a suitable regexp
matching the DB name in the env var
  OSSTEST_DB_USEREAL_IGNORETEST

This means that when a test database is in play, the user who created
it cannot accidentally operate on the real DB.

The safety catch does not affect Tcl programs, which get the DB config
directly, but in general that just means sg-execute-flight and
sg-run-job which already have a fair amount of safety catch because
they demand flight numbers.

mg-schema-test-database hits this feature over the head.  We assume
that the caller of mg-schema-test-database knows what they are doing;
particularly, that if they create nested test DBs (!), they do not
need the assitance of this feature to stop themselves operating
mg-schema-test-database incorrectly.  Anyone who creates nested test
DBs will hopefully recognise the potential for confusion!

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Fix unclarity in a comment.

9 years agomg-schema-test-database: Bump flight sequence number in test DB
Ian Jackson [Mon, 7 Dec 2015 17:22:54 +0000 (17:22 +0000)]
mg-schema-test-database: Bump flight sequence number in test DB

This makes test flights have different numbers to those currently in
production, which will help avoid accidents.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch

9 years agomg-schema-test-database: Change username for back-to-main-db xref
Ian Jackson [Mon, 7 Dec 2015 16:37:03 +0000 (16:37 +0000)]
mg-schema-test-database: Change username for back-to-main-db xref

The `username' of the xdbref task in the test db, referring to the
main db, is changed to `PARENT' (from `<username>@<nodename>').

Currently this is purely cosmetic, but it is going to be useful to
distinguish the two cases:
 * This is a test DB and contains references to a parent
 * This is a parent DB (probably the main DB) which contains
   references to child test DB(s).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Fix `DBS' in commit message to `DB(s)'.
v2: New patch

9 years agoConfiguration: Introduce $c{Username}
Ian Jackson [Mon, 7 Dec 2015 16:03:40 +0000 (16:03 +0000)]
Configuration: Introduce $c{Username}

This makes it easier to share the output of whoami.  As a beneficial
side effect it can now be overridden.

Replace many open-coded calls to `whoami` etc. with references to
$c{Username}.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: New patch

9 years agomg-schema-test-database: Sort out daemons; provide `daemons' subcommand
Ian Jackson [Fri, 4 Dec 2015 18:24:44 +0000 (18:24 +0000)]
mg-schema-test-database: Sort out daemons; provide `daemons' subcommand

We arrange for the test configuration to look for the daemons on a
different host and port, and we provide a convenient way to run such a
pair of daemons.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Moved setting of *Daemon{Host,Port} to this patch (was
     previously in `mg-schema-test-database: New script')

9 years agomg-schema-test-database: Move setting of test_cfg_setting to dbname
Ian Jackson [Fri, 4 Dec 2015 19:13:08 +0000 (19:13 +0000)]
mg-schema-test-database: Move setting of test_cfg_setting to dbname

This will makes it available to a wider subset of the script, which is
going to be important in a moment.

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agomg-schema-test-database: New script
Ian Jackson [Fri, 4 Dec 2015 17:57:54 +0000 (17:57 +0000)]
mg-schema-test-database: New script

This allows a user in non-standalone mode to make a whole new test
database, which is largely a clone of the original database.

The new db refers to the same resources (hosts), and more-or-less
safely borrows some of those hosts.

Currently we don't do anything about the queue and owner daemons.
This means that queue-daemon-based resource allocation is broken when
clients are pointed at the test db.  But non-queue-based allocation
(eg, ./mg-allocate without -U) works, and the test db can be used for
db-related experiments and even support individual ts-* scripts (other
than ts-hosts-allocate of course).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Do not set *Daemon{Host,Port} - move this chunk to a later patch

9 years agoOsstest.pm: Break out and export globalconfigfiles
Ian Jackson [Fri, 4 Dec 2015 18:06:09 +0000 (18:06 +0000)]
Osstest.pm: Break out and export globalconfigfiles

No functional change; no callers as yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocri-getconfig: Provide debugging for get_psql_cmd
Ian Jackson [Wed, 25 Nov 2015 15:34:04 +0000 (15:34 +0000)]
cri-getconfig: Provide debugging for get_psql_cmd

This allows us to execute only the first <some number> SQL
invocations.  The first non-executed one is dumped, instead, by having
get_psql_command print a rune involving ./mg-debug-fail (which the
caller will then execute).

The locking makes things work roughly-correctly if get_psql_cmd is run
in multiple processes at once: it is not defined exactly which
invocations get which counter values, but they will all work properly
and get exactly one counter value each.

If set -x is in force, turn it off for get_psql_cmd: our perl rune is
uninteresting to see repeated ad infinitum in debugging output.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocri-getconfig: Provide get_psql_cmd and get_pgdump_cmd
Ian Jackson [Fri, 4 Dec 2015 18:03:30 +0000 (18:03 +0000)]
cri-getconfig: Provide get_psql_cmd and get_pgdump_cmd

This is for (non-standalone-mode) shell scripts which want to access
the postgresql database.

get_psql_command provides `-v ON_ERROR_STOP' because it is not the
default (!) and no sane caller would not want it.

No callers as yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix typo in comment.

9 years agomg-debug-fail: Catch attempts to read from a tty
Ian Jackson [Fri, 4 Dec 2015 18:41:10 +0000 (18:41 +0000)]
mg-debug-fail: Catch attempts to read from a tty

When stdin is a tty, do not try to dump it.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agomg-debug-fail: New utility script for debugging
Ian Jackson [Fri, 4 Dec 2015 18:12:38 +0000 (18:12 +0000)]
mg-debug-fail: New utility script for debugging

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Use "egrep ''" rather than "egrep .".  Both sanitise
     missing-final-newline but "egrep ''" will print blank lines,
     which is desirable here.

9 years agoConfiguration: No longer set password=<~/.xen-osstest/db-password>
Ian Jackson [Fri, 4 Dec 2015 18:00:48 +0000 (18:00 +0000)]
Configuration: No longer set password=<~/.xen-osstest/db-password>

Instead, expect the user to provide ~/.pgpass.

This is a good idea because we don't really want to be handling
passwords ourselves if we can help it.  And, we are shortly going to
want to do some exciting mangling of the database access
configuration, which would be complicated by the presence of this
password expansion.

This may break for some users of existing Executive (non-standalone)
setups which are using production-config-cambridge or the default
built-in configuration.

DEPLOYMENT NOTE: After this passes the push gate in Cambridge,
/export/home/osstest/.{xen-,}osstest/db-password should be deleted to
avoid confusion in the future.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocri-getconfig: Break out exec_resetting_sigint.
Ian Jackson [Fri, 4 Dec 2015 19:11:41 +0000 (19:11 +0000)]
cri-getconfig: Break out exec_resetting_sigint.

Move this oddity (and the associated comment) from
standalone-generate-dump-flight-runvars to cri-getconfig.  We are
going to want it elsewhere.

We put this in cri-getconfig because that is the one library of
generic shell functions which everything includes.  Perhaps this file
is misnamed.

No overall functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agotcl daemons: log host and port number we bind to, at startup
Ian Jackson [Fri, 4 Dec 2015 19:03:16 +0000 (19:03 +0000)]
tcl daemons: log host and port number we bind to, at startup

If the socket setup fails, this makes it easier to see what the
program was trying to do.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocr-try-bisect-adhoc: Set OSSTEST_PRIORITY=-30
Ian Campbell [Thu, 3 Dec 2015 14:55:48 +0000 (14:55 +0000)]
cr-try-bisect-adhoc: Set OSSTEST_PRIORITY=-30

This makes adhoc bisects slightly more important than smoke tests, on
the basis that a smoke test can choose another host while an adhoc
bisect cannot.

Document this is README.planner and while there make a note of the
usage of OSSTEST_RESOURCE_WAITSTART by cr-try-bisect.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoproduction-config-cambridge: Use new squid proxy
Ian Jackson [Fri, 4 Dec 2015 13:57:59 +0000 (13:57 +0000)]
production-config-cambridge: Use new squid proxy

Specify both HttpProxy and DebianMirrorProxy.  In my tests this seems
to improve some of the apparently-intercepting-proxy-related failures,
and it will certainly improve logging.

I set DebianMirrorProxy too so that queries to security.d.o go through
the proxy.  Ideally we would have a apt cache that could be used as an
http proxy rather than as an origin server; when that happens we can
set DebianMirrorProxy to point to it and do away with DebianMirrorHost
(as we do in Massachusetts).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocr-try-bisect-adhoc: Add a reminder about OSSTEST_EMAIL_HEADER
Ian Campbell [Thu, 3 Dec 2015 15:35:30 +0000 (15:35 +0000)]
cr-try-bisect-adhoc: Add a reminder about OSSTEST_EMAIL_HEADER

... I'm forever spamming Ian J for a couple of iterations when I set
one of these up and forget to override the default destination.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agocr-try-bisect-adhoc: Set laundered_testid so graph URL is correct
Ian Campbell [Wed, 2 Dec 2015 16:05:04 +0000 (16:05 +0000)]
cr-try-bisect-adhoc: Set laundered_testid so graph URL is correct

Otherwise the testid is missing from the filename, resulting in e.g.
http://osstest.test-lab.xenproject.org/~osstest/pub/results-adhoc/bisect/xen-unstable/test-amd64-amd64-qemuu-nested-intel..svg

Instead of test-amd64-amd64-qemuu-nested-intel.debian-hvm-install-l1-l2.svg

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agocr-try-bisect-adhoc: Ensure tmp exists.
Ian Campbell [Wed, 2 Dec 2015 15:58:58 +0000 (15:58 +0000)]
cr-try-bisect-adhoc: Ensure tmp exists.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agosg-run-job: Coalesce a couple of repetitions
Ian Jackson [Fri, 27 Nov 2015 16:36:23 +0000 (16:36 +0000)]
sg-run-job: Coalesce a couple of repetitions

Fold `guest-localmigrate.2' into `guest-localmigrate/x10' and move
`guest-start.2' to after `guest-start.repeat' (reversing the contents
of the latter so that the start comes before the stop).
(guest-start.2 is still necessary because the start/stop test leaves
the guest stopped, whereas the subsequent destroy test ought happen
with the guest running.)

This change will allow the heisenbug compensator to see more of these
failures as the same failures.

The overall effect includes a reduction of the number of localhost
migrations from 11 to 10, but this is better than leaving a misleading
testid containing the string `x10' (or changing the testid).

It is best to fold this way, keeping the testid of the step which
previously had most of the regressions, because: the alternative,
keeping the testid of the low-repetition step, would allow osstest to
use previous lucky passes of the low-repetition step to justify
current failures of the now-high-repetition step.

To check that the effect of the patch is as intended, I ran a before
and after run with OSSTEST_SIMULATE=1, and (a) collected and sedded
and diffed the sg-run-job transcripts and (b) looked in the db.

I also ran a real test (65261 in the Xen Project test lab) with a very
similar version, which passed, and will re-run that before pushing.

(a):

  c&p transcripts from mg-execute-flight email reports
  perl -i~ -pe 's/\b(38371|38370|65261|38395|38397)\b/FLIGHT/; s/^2015-11-\d\d \S+ /TIME /' [tu]
  diff -u [tu] >v
  grep starting v

 =>

 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-build-check  build-check(1)
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-saverestore host debian guest-saverestore.2
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-localmigrate host debian guest-localmigrate.2
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-localmigrate x10 host debian guest-localmigrate/x10
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-stop host debian guest-stop
+TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-repeat-test 10 ts-guest-start host debian {;} ts-guest-stop host debian guest-start/debian.repeat
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-start host debian guest-start.2
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-repeat-test 10 ts-guest-stop host debian {;} ts-guest-start host debian guest-start/debian.repeat
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-guest-destroy host debian guest-destroy
 TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-leak-check check host leak-check/check
-TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-capture host capture-logs(24)
+TIME Z [test-amd64-i386-xl] starting FLIGHT.test-amd64-i386-xl ts-logs-capture host capture-logs(23)

(b)

osstestdb=> select * from (select job,stepno,step,status,testid from steps where flight=38370) before full outer join (select job,stepno,step,status,testid from steps where flight=38400) after using (testid) order by coalesce(before.stepno, after.stepno);
          testid           |        job         | stepno |             step             | status |        job         | stepno |             step             | status
---------------------------+--------------------+--------+------------------------------+--------+--------------------+--------+------------------------------+--------
 build-check(1)            | test-amd64-i386-xl |      1 | ts-build-check               | pass   | test-amd64-i386-xl |      1 | ts-build-check               | pass
 hosts-allocate            | test-amd64-i386-xl |      2 | ts-hosts-allocate            | pass   | test-amd64-i386-xl |      2 | ts-hosts-allocate            | pass
 host-install(3)           | test-amd64-i386-xl |      3 | ts-host-install-twice        | pass   | test-amd64-i386-xl |      3 | ts-host-install-twice        | pass
 host-ping-check-native    | test-amd64-i386-xl |      4 | ts-host-ping-check           | pass   | test-amd64-i386-xl |      4 | ts-host-ping-check           | pass
 xen-install               | test-amd64-i386-xl |      5 | ts-xen-install               | pass   | test-amd64-i386-xl |      5 | ts-xen-install               | pass
 xen-boot                  | test-amd64-i386-xl |      6 | ts-host-reboot               | pass   | test-amd64-i386-xl |      6 | ts-host-reboot               | pass
 host-ping-check-xen       | test-amd64-i386-xl |      7 | ts-host-ping-check           | pass   | test-amd64-i386-xl |      7 | ts-host-ping-check           | pass
 leak-check/basis(8)       | test-amd64-i386-xl |      8 | ts-leak-check                | pass   | test-amd64-i386-xl |      8 | ts-leak-check                | pass
 debian-install            | test-amd64-i386-xl |      9 | ts-debian-install            | pass   | test-amd64-i386-xl |      9 | ts-debian-install            | pass
 debian-fixup              | test-amd64-i386-xl |     10 | ts-debian-fixup              | pass   | test-amd64-i386-xl |     10 | ts-debian-fixup              | pass
 guest-start               | test-amd64-i386-xl |     11 | ts-guest-start               | pass   | test-amd64-i386-xl |     11 | ts-guest-start               | pass
 migrate-support-check     | test-amd64-i386-xl |     12 | ts-migrate-support-check     | pass   | test-amd64-i386-xl |     12 | ts-migrate-support-check     | pass
 saverestore-support-check | test-amd64-i386-xl |     13 | ts-saverestore-support-check | pass   | test-amd64-i386-xl |     13 | ts-saverestore-support-check | pass
 guest-saverestore         | test-amd64-i386-xl |     14 | ts-guest-saverestore         | pass   | test-amd64-i386-xl |     14 | ts-guest-saverestore         | pass
 guest-localmigrate        | test-amd64-i386-xl |     15 | ts-guest-localmigrate        | pass   | test-amd64-i386-xl |     15 | ts-guest-localmigrate        | pass
 guest-saverestore.2       | test-amd64-i386-xl |     16 | ts-guest-saverestore         | pass   | test-amd64-i386-xl |     16 | ts-guest-saverestore         | pass
 guest-localmigrate.2      | test-amd64-i386-xl |     17 | ts-guest-localmigrate        | pass   |                    |        |                              |
 guest-localmigrate/x10    | test-amd64-i386-xl |     18 | ts-guest-localmigrate        | pass   | test-amd64-i386-xl |     17 | ts-guest-localmigrate        | pass
 guest-stop                | test-amd64-i386-xl |     19 | ts-guest-stop                | pass   | test-amd64-i386-xl |     18 | ts-guest-stop                | pass
 guest-start.2             | test-amd64-i386-xl |     20 | ts-guest-start               | pass   | test-amd64-i386-xl |     20 | ts-guest-start               | pass
 guest-start/debian.repeat | test-amd64-i386-xl |     21 | ts-repeat-test               | pass   | test-amd64-i386-xl |     19 | ts-repeat-test               | pass
 guest-destroy             | test-amd64-i386-xl |     22 | ts-guest-destroy             | pass   | test-amd64-i386-xl |     21 | ts-guest-destroy             | pass
 leak-check/check          | test-amd64-i386-xl |     23 | ts-leak-check                | pass   | test-amd64-i386-xl |     22 | ts-leak-check                | pass
 capture-logs(23)          |                    |        |                              |        | test-amd64-i386-xl |     23 | ts-logs-capture              | pass
 capture-logs(24)          | test-amd64-i386-xl |     24 | ts-logs-capture              | pass   |                    |        |                              |
(25 rows)

osstestdb=>

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Do not increment count of migration tests so as to make
     testid misleading.
    Do the change to the start/stop test differently.

9 years agosg-run-job: Unroll test-guest-migr iteration loop
Ian Jackson [Fri, 27 Nov 2015 16:34:39 +0000 (16:34 +0000)]
sg-run-job: Unroll test-guest-migr iteration loop

We are going to want to do something different on the 2nd iteration,
so do away with the loop.  No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agocs-bisection-step: Limit size of revision log included in reports
Ian Jackson [Mon, 30 Nov 2015 13:35:54 +0000 (13:35 +0000)]
cs-bisection-step: Limit size of revision log included in reports

There is a limit in cr-daily-branch, but none in cs-bisection-step.

adhoc-revtuple-generator could usefully have this built in but that's
not so simple, so do it again here.  We already slurp the whole thing
into core so from a resource usage point of view we might as well do
the length check here too.

Reported-by: David Vrabel <david.vrabel@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix typo in message.

9 years agoREADME.email: Add `Worked example of relevant regression in previous flight'
Ian Jackson [Fri, 27 Nov 2015 15:37:59 +0000 (15:37 +0000)]
README.email: Add `Worked example of relevant regression in previous flight'

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoREADME.email: Document `fail in 58948 REGR. vs. 63449'
Ian Jackson [Fri, 27 Nov 2015 11:36:05 +0000 (11:36 +0000)]
README.email: Document `fail in 58948 REGR. vs. 63449'

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Change an example number to improve clarity.

9 years agoExecutive HTML output: More varieties of grey fruit, fewer bananas
Ian Jackson [Wed, 25 Nov 2015 11:38:39 +0000 (11:38 +0000)]
Executive HTML output: More varieties of grey fruit, fewer bananas

The use of yellow for `preparing' and `running' is particularly
inappropriate in the projection summary, but is also rather misleading
when showing cancelled flights.

Use shades of grey for the different levels of in-progress-ness.
Darker shades are `more running', which seems to align better with the
shades in `Scheduled' in the flight summary.

Yellow now definitely means something is `broken', or worse.

The two places where this needs to be changed are actually
meaningfully different: report_run_getinfo works on job statuses,
whereas sg-report-flight handles only steps, which have many fewer
statuses.

Here are some samples of the ouput, from the Citrix Cambridge
instance:

 http://xenbits.xen.org/people/iwj/2015/flights-summary-2.html
 http://xenbits.xen.org/people/iwj/2015/sg-report-flight/
 http://xenbits.xen.org/people/iwj/2015/sg-report-flight/test-amd64-amd64-xl-rtds/info.html

Despite using more different shades, the result is now both somewhat
less fruity looking and IMO easier to make sense of.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agots-debian-di-install: Don't set runvars for netboot kernel+ramdisk as outputs
Ian Campbell [Mon, 30 Nov 2015 11:58:42 +0000 (11:58 +0000)]
ts-debian-di-install: Don't set runvars for netboot kernel+ramdisk as outputs

Currently these runvars are either URLs provided by the definition
(e.g. make-flight) or output controller-relative paths created by the
execution (in the case where they aren't from the definition).

This wierd dual-semantics is confusing and wrong, and in particular is
broken if the test step is rerun (e.g. in standalone mode).

In the case where they are outputs only  these paths is information
only. The information is already available in the full logs so
dropping the runvars here merely removes the information from the
summary table. It's not so useful that this is an issue.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agotarget_fetchurl: Handle undefined $c{HttpProxy}
Ian Campbell [Mon, 30 Nov 2015 11:48:31 +0000 (11:48 +0000)]
target_fetchurl: Handle undefined $c{HttpProxy}

Avoiding a usage of a potentially undefined variable.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomake-flight: move in-lined branch vs arch filtering into callbacks
Ian Campbell [Tue, 17 Nov 2015 17:13:46 +0000 (17:13 +0000)]
make-flight: move in-lined branch vs arch filtering into callbacks

No change to the output of standalone-generate-dump-flight-runvars

The inlined xenbranch vs arch filters remain where they are since they
are common (in that they reflect the addition and removal of arches)
and apply equally to all make-*-flight.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomake-flight: consolidate branch_filter_callback for builds and tests
Ian Campbell [Tue, 17 Nov 2015 17:13:45 +0000 (17:13 +0000)]
make-flight: consolidate branch_filter_callback for builds and tests

Currently we have a test_matrix_branch_filter_callback which filters
jobs based on the $xenarch and $branch and a separate more adhoc
filter inline for the build jobs.

This has lead to things getting out of sync in the past (e.g. recently
we dropped armhf tests from the linux-3.10 and -3.14 branches but not
the build jobs).

Add a new build_matrix_branch_filter_callback and for make-flight
cause this and test_matrix_branch_filter_callback to use a common
helper.

The adhoc filtering in the build loop remains and will be tidied up
next.

For make-distros-flight just add a nop build filter alongside the test
filter.

No change to the output of standalone-generate-dump-flight-runvars.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomake-*flight: Reorder.
Ian Campbell [Tue, 17 Nov 2015 17:13:44 +0000 (17:13 +0000)]
make-*flight: Reorder.

I just got tripped up again by putting a build job filter definition
after the call to create_build_jobs. Reorder the make-*flight scripts
to reduce the probability of me doing so any more times.

The general order of these scripts is now:
  - job filter callbacks
  - test job creation
  - top-level code which drives the process.

No change to the output of standalone-generate-dump-flight-runvars.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomfi-common: Drop armhf build jobs on older linux-* branches
Ian Campbell [Tue, 17 Nov 2015 17:13:43 +0000 (17:13 +0000)]
mfi-common: Drop armhf build jobs on older linux-* branches

test_matrix_branch_filter_callback was recently updated to exclude
testing of linux-3.10 (54f237784d4b) and 3.14 (b0c5663a03e7), however
this only excludes test and not build jobs so we were still trying to
build the arm kernel there.

This changes the build job filtering to be in sync with the test job
filtering, the net result is to remove build-armhf* from linux-3.10
and linux-3.14.

A subsequent patch will try and combine those two filters into one to
prevent this skew happening again.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agomg-adjust-flight-makexrefs: Make a nop in Standalone mode
Ian Campbell [Tue, 17 Nov 2015 17:13:42 +0000 (17:13 +0000)]
mg-adjust-flight-makexrefs: Make a nop in Standalone mode

It is not useful to try and drop jobs based on scrobbling around in
the standalone history.

This makes it possible to "./standalone make-flight
xen-unstable-smoke" and fixes standalone-generate-dump-flight-runvars

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoDebian: Move FANCYTTY=0 setting from preseed_create to preseed_base
Ian Campbell [Tue, 24 Nov 2015 15:47:09 +0000 (15:47 +0000)]
Debian: Move FANCYTTY=0 setting from preseed_create to preseed_base

This makes the console logs of any HVM or debian-installer created
easier to parse by omitting the escape characters.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoDebian: Move sysvinit-core install from preseed_create to preseed_base
Ian Campbell [Tue, 24 Nov 2015 15:39:00 +0000 (15:39 +0000)]
Debian: Move sysvinit-core install from preseed_create to preseed_base

preseed_create is used only for physical host installs, but we want
this workaround to apply to at least ts-debian-hvm-install'd guests
which are going to be used for nested testing.

I can't see any harm in doing this globally for all Debian HVM guests,
as well as PV guests installed using Debian Installer.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoSet {ident}_suite runvar when install a Debian guest.
Ian Campbell [Mon, 23 Nov 2015 11:54:11 +0000 (11:54 +0000)]
Set {ident}_suite runvar when install a Debian guest.

Currently those places which want this open code a lookup of the
{ident}_suite runvar with a fallback to the configuration file.

However selecthost was missing such a lookup in the case where it is
constructing a nested L1 host (which begins from the selectguest
template), which lead to ts-xen-install on Jessie missing the
installation of libnl-route-3-200.

Fix this by providing debian_guest_suite($gho) which as well as
initialising $gho->{Suite} stores an {ident}_suite runvar (taking care
to handle the case where one is already set by e.g. make-flight). For
convenience debian_guest_suite() also returns the suite name.

ts-debian-install, ts-debian-di-install and ts-debian-hvm-install now
use debian_guest_suite instead of open coding the lookup.

The final piece of the puzzle is to have selectguest() pickup the
{ident}_suite runvar (if it is set) and initialise $gho->{Suite} from
it.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agosg-run-job: Make nested-layer-descend insist on $ok
Ian Jackson [Mon, 23 Nov 2015 15:53:53 +0000 (15:53 +0000)]
sg-run-job: Make nested-layer-descend insist on $ok

per-host-ts does nothing if !$ok, so per-host-prep's individual steps
become no-ops if the host prep fails, which prevents blundering on.
But per-host-prep does not throw.

The other call site explicitly avoids calling the recipe script if
!$ok.  nested-layer-descend is for calling from within a recipe, so we
need to throw an exception to abort the script, if !$ok.

Reported-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agomake-flight: Run separate nested jobs on AMD and Intel
Ian Campbell [Mon, 23 Nov 2015 10:49:26 +0000 (10:49 +0000)]
make-flight: Run separate nested jobs on AMD and Intel

nested HVM relies heavily on the underlying HVM implementation, which
is different for Intel and AMD and therefore worth testing separately.

Currently test-amd64-amd64-qemuu-nested is not tied to any specific
vendor, split it into -amd and -intel jobs and set the host flags
appropriately.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: "Hu, Robert" <robert.hu@intel.com>
9 years agoms-queuedaemon: Update projection reports when idle
Ian Jackson [Wed, 18 Nov 2015 14:39:00 +0000 (14:39 +0000)]
ms-queuedaemon: Update projection reports when idle

When the planning queue becomes empty, we update the plan.  We should
also update the projection.

I have cowboyed this change in the Cambridge instance's
daemons-testing.git and tested it there.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agomake-flight: Stop testing ARM for linux-3.14 flights
Ian Campbell [Tue, 17 Nov 2015 10:58:39 +0000 (10:58 +0000)]
make-flight: Stop testing ARM for linux-3.14 flights

Since d4dba6183d61 "ts-kernel-build: Include dtbs in dist file" we
know require a "make dtbs_install" target, which was only added after
3.14.

None of the ARM h/w in the XenProject test colo can run with a kernel
this old. We do have a system in the Citrix instance in Cambridge
which does but the complexity vs benefit of testing this doesn't
warrant continuing to test 3.14.

Our current baseline for other ARM testing is a 3.16 based
linux-arm-xen branch.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
9 years agostandalone: only rotate logs if savelog is available
Ian Campbell [Wed, 14 Oct 2015 09:45:36 +0000 (10:45 +0100)]
standalone: only rotate logs if savelog is available

`savelog' comes from the `debianutils' package and so is unlikely to
be available elsewhere. Revert to the old behaviour of clobbering the
logs in this case.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agots-logs-capture: Attempt to power up a host before capturing its logs
Ian Jackson [Mon, 16 Nov 2015 15:46:10 +0000 (15:46 +0000)]
ts-logs-capture: Attempt to power up a host before capturing its logs

If for any reason it's powered down, this is useful.

(In theory for a nested host, we could get the logs by grobbling in
its parent's view of its filesystem, but that is too much work.)

We do not trap the errors from this, so that if the PDU is broken (or
the L1 has not been created), we can fail more quickly.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoNested: Make Osstest::PDU::guest idempotent
Ian Jackson [Mon, 16 Nov 2015 15:44:11 +0000 (15:44 +0000)]
Nested: Make Osstest::PDU::guest idempotent

These methods are supposed to succeed, silently, if the host is
already in the required power state.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoNested: Do not attempt to look up nested host ownership in DB
Ian Jackson [Mon, 16 Nov 2015 15:32:25 +0000 (15:32 +0000)]
Nested: Do not attempt to look up nested host ownership in DB

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agologm: Introduce $logm_prefix, and annotate nested guest messages
Ian Jackson [Mon, 16 Nov 2015 15:21:12 +0000 (15:21 +0000)]
logm: Introduce $logm_prefix, and annotate nested guest messages

This allows code elsewhere to annotate log messages which might
otherwise be confusing.  The variable should be localised, and the
value should always start with a space, if not empty.

Use this to annotate the calls to selecthost and selectguest from
within selecthost-for-an-L1.  Otherwise some of the log messages can
be very confusing.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoLog capture pathnames: Include host name (and nested L0 name)
Ian Jackson [Mon, 16 Nov 2015 12:33:51 +0000 (12:33 +0000)]
Log capture pathnames: Include host name (and nested L0 name)

Rather than using $gho->{Name} (and, sometimes, separately
$ho->{Name}), use hostnamepath().

This means that the hostname is always included in the standard way.
The filename structure for xenctx and console snapshots changes
slightly.

In two cases this means that the hostname is now included where it
wasn't previously.  That is helpful if due to some insanity the same
guest is present on more than one host.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoNested hosts: Use full hostnamepath in various paths
Ian Jackson [Mon, 16 Nov 2015 17:50:15 +0000 (17:50 +0000)]
Nested hosts: Use full hostnamepath in various paths

Replace some (but not all) interpolations of $ho->{Name} in filenames
with hostnamepath($ho).

I searched the tree for `{Host}' and `{Name}', looking for paths which
need (or might need) to be qualified with an L0 hostname in an L1 or
L2 case.

I have left ts-logs-capture for now; that will come in a moment.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoNested hosts: Print L0 as well as L1 parent name in selectguest
Ian Jackson [Mon, 16 Nov 2015 12:26:11 +0000 (12:26 +0000)]
Nested hosts: Print L0 as well as L1 parent name in selectguest

Now that we have hostnamepath_list, we can remove the ad-hoc
expansion "$gn on $...{Name}" with a recipe which ascends through the
applicable nesting levels.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoms-flights-summary: Improve the overview table
Ian Jackson [Tue, 17 Nov 2015 11:48:50 +0000 (11:48 +0000)]
ms-flights-summary: Improve the overview table

- Align the queued/preparing/running/etc. totals into table columns.
- Use <strong> to highlight `queued' jobs.
- Use <strong> to highlight the end time if it is the flight end,
  not just a phase end.
- No longer print the `total unqueued' separately.
- Show the `common info'.

Replace much of the HTML generation with plain literal or
almost-literal HTML, since the HTML is complex to generate but easy to
write.

An example of the output can be seen here:
   http://xenbits.xen.org/people/iwj/2015/flights-summary.html

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoNested hosts: Use hostnamepath() in create_webfile
Ian Jackson [Mon, 16 Nov 2015 12:19:54 +0000 (12:19 +0000)]
Nested hosts: Use hostnamepath() in create_webfile

create_webfile needs a pathname in the shared public-html directory.
These paths need to be (a) stable (b) unique across all running jobs.
We achieve this by basing the filenames on the hostname and (for a
guest) the guest name.

But for an L2 guest we need to include the physical host name too,
because the L1 `host' is not unique.

Fix this by using hostnamepath(), replacing the open-coded single
iteration.

Reported-by: Ian Campbell <ian.campbell@citrix.com>
CC: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoNested hosts: Provide hostnamepath and hostnamepath_list
Ian Jackson [Mon, 16 Nov 2015 12:17:40 +0000 (12:17 +0000)]
Nested hosts: Provide hostnamepath and hostnamepath_list

This can (and often should) be used to replace $ho->{Name}.

For an L0 host it returns "$ho->{Name}", ie HOST.

For a plain guest or L1 guest it returns
"$ho->{Host}{Name}_$ho->{Name}", ie HOST_GUEST or HOST_L1.

For an L2 guest it recurses further, giving HOST_L1_L2.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoSerial::xenuse: Send xenuse output to /dev/null
Ian Jackson [Wed, 11 Nov 2015 11:51:37 +0000 (11:51 +0000)]
Serial::xenuse: Send xenuse output to /dev/null

Like sympathy, attaching via xenuse causes xenuse to send output from
the host to its own stdout.

But we don't want the ts-logs-capture stdout to contain this serial
output, interleaved with its own log messages.  We'll capture the
whole serial log from the xenuse logfile.  So redirect it to /dev/null.

I have checked that xenuse does (at least sometimes, eg when given a
nonexistent hostname) use stderr when something actually goes wrong.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: New patch

9 years agoSerial: Add new serial method object for `guest' type
Robert Ho [Mon, 2 Nov 2015 09:21:13 +0000 (17:21 +0800)]
Serial: Add new serial method object for `guest' type

L1 guests' serial ports are owned by qemu in L0.  We can send them
debug keys by writing to the qemu pipe.

(xl debug-key looks like it would be useful but it actually sends
debug keys to the hypervisor of the host it is running on.  We want to
send the debug keys to the hypervisor and kernel from the outside.)

Log fetching is not needed because from the POV of the L0 the L1 is a
guest, so the L0's log capture will already fetch the L1's serial
console output.

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Get FIFO path right.
     Log the ssh command.
     Put quotes around the redirection, as required.
     Export sshuho from TestSupport.
     Make ::guest->fetch_logs actually use @_ so that it works.
v16: Mostly rewritten: now uses new keys_real base class, and
     uses the qemu pipe rather than xl debug-keys.
v15: New patch.

9 years agoSerial: Factor out Osstest::Serial::keys_real
Ian Jackson [Tue, 10 Nov 2015 19:11:01 +0000 (19:11 +0000)]
Serial: Factor out Osstest::Serial::keys_real

The sympathy and xenuse serial modules had too much in common.  Factor
out the common code, which is now responsible for
  - knowledge of the Xen console switch
  - splitting strings up into individual keys
  - timing decisions
  - error trapping and logging

This new class is an abstract base class for the concrete serial
method classes, and calls back to its derived class to prepare, send
each actual key, and shut down.

There is some functional change: notably, after failure to send the
first debug key, sending the remainder will not be attempted.

While we're here, fix a typo `dettach' to `detach'.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Move `force attach' and `force detach' writes into xenuse.pm
      where they belong.
     Add typo fix.

9 years agoHVM guests: Use qemu "pipe:" for serial output logging
Ian Jackson [Tue, 10 Nov 2015 18:00:03 +0000 (18:00 +0000)]
HVM guests: Use qemu "pipe:" for serial output logging

Modern qemu has the "pipe:/PATH" character driver.  This opens
/PATH.in for reading and /PATH.out for writing.  In my tests, I found
that:
  - contrary to the documentation, they do not need to be pipes
    (at least, /PATH.out can be a file)
  - but they must both already exist
  - qemu will follow symlinks, so /PATH.out can be a symlink to
    a file
  - if /PATH.in is a fifo, qemu will tolerate other processes opening
    it for writing, and writing things, only occasionally.  (Probably,
    qemu opens it O_RDWR; or perhaps it reopens it after EOF.)

Use this feature to achieve the following:
  - guest serial output ends up in /var/log/xen/osstest-serial-GUEST.log
    (which is already captured by ts-logs-capture) rather than
    interleaved with qemu's stderr output (in the libxl-created logfile)
  - guest serial input comes from a pipe in /root which we can open
    and write to if we want to talk to the guest

We are mostly interested in the final bullet point, because that will
allow us to send debug keys to the emulated serial port of an L1
nested HVM guest.

Looking at the source code of qemu in 4.2 and 4.6 I think the above
approach will work with all relevant qemu-xen's.

If the device model version is qemu-xen-traditional, `pipe:' is not
supported.  If device_model_version is not set, we will be using
whatever the xen.git we used defaults to.  For Xen 4.1 and earlier
that is qemu-xen-traditional, and I'm slightly loathe to break osstest
for those earlier versions.  There doesn't seem to be anything else in
the runvars that would clue us in.  So be cautious and do not use the
new feature unless device_model_version is explicitly set.

The nested tests are all -qemuu so set device_model_version.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Fix fifo mode to be 600, not 700
v16: New patch

9 years agoOsstest/Testsupport.pm: use get_target_property() for some host setup
Robert Ho [Sat, 31 Oct 2015 05:42:03 +0000 (13:42 +0800)]
Osstest/Testsupport.pm: use get_target_property() for some host setup

For nested cases, nested host can inherit its host's property for
dhcp watch setup and ether_prefix property setup.

Signed-off-by: Robert Ho <robert.hu@intel.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v15: New patch

9 years agoOsstest/Testsupport.pm: change target's default kernkind to 'pvops'
Ian Jackson [Sat, 31 Oct 2015 05:35:26 +0000 (13:35 +0800)]
Osstest/Testsupport.pm: change target's default kernkind to 'pvops'

This is safe only if no existing flights would be affected.  (That is,
the meaning of no existing sets of runvars would be changed.)

To check whether this would make any difference I did some database
searches.  Since any time target_kernkind_check is called it sets a
corresponding `console' runvar, I can search for `console' without a
corresponding `kernkind'.  I ran this query:

  select * from (select *, (select name from runvars r2 where
  r2.flight=r1.flight and r2.job=r1.job and r2.name=
  replace(r1.name,'console','kernkind')) kk from runvars r1 where
  r1.name like '%console') iq where kk is null order by flight desc;

and it found nothing since flight 7682.  So I think we can change the
default.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v15: New patch

9 years agots-xen-install: networking: Rename `nodhcp' to `ensurebridge'
Ian Jackson [Thu, 10 Sep 2015 17:09:16 +0000 (18:09 +0100)]
ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'

This function does not (now) always undo the DHCP configuration.
Sometimes it leaves it.  Its main function is to ensure that we have
a bridge for use by guests.

So rename the function.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: This patch was previously 4/4 of a miniature series containing
      a different way of dealing with the Nested HVM L1 DHCP problem.

9 years agots-xen-install: Properly handle hosts without a static IP address
Robert Ho [Fri, 14 Aug 2015 03:55:50 +0000 (11:55 +0800)]
ts-xen-install: Properly handle hosts without a static IP address

Check IpStatic, and if it is not set, provide a dhcp stanza in
/etc/network/interfaces, rather than an `inet static' one.

This is necessary for L1 nested hosts, because they don't have a
static IP address.

In principle this makes matters more correct for physical hosts
without static IP addresses, but these are currently not supported
by selecthost().

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Only use `inet dhcp' if !$ho->{IpStatic}.

9 years agoNested HVM: Add test job to appropriate flights
Robert Ho [Mon, 17 Aug 2015 08:40:22 +0000 (16:40 +0800)]
Nested HVM: Add test job to appropriate flights

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
---
v17: Use usual_debianhvm_image
v14: Use default gueststorage_size, rather than setting runvar.
     Dropped acked from Ian Campbell.

9 years agoNested HVM: Provide test-nested recipe
Robert Ho [Mon, 17 Aug 2015 09:07:02 +0000 (17:07 +0800)]
Nested HVM: Provide test-nested recipe

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: ts-nested-setup command line syntax updated.
v15: (Robert Ho) remove the unnecessary l1 destroy; as it will
     implicitly powered off by framework as a nested host.
     This change hasn't been confirmed by Ian Jackson yet; I
     plan to separate it as fix patch but by mistake squashed
     it in. Ian Jackson may want to revert this if he dosn't
     agree.

9 years agoNested HVM: Provide ts-nested-setup to help make L1 usable as a host
Robert Ho [Mon, 17 Aug 2015 07:58:01 +0000 (15:58 +0800)]
Nested HVM: Provide ts-nested-setup to help make L1 usable as a host

* Provide the L1 with some storage for its own guests' disks
* Install some packages in the L1
* Optionally, set a runvar defining the L1 for the rest of the job

The recipe is going to run ts-xen-install etc.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Use target_check_ip (renamed, earlier, in a new patch)
     ts-nested-setup now has a default gueststorage_size of 20G
      and this is implicitly used by make-flight.
     Adjusted for new selecthost nested host syntax and correspondingly
      completely changed invocation syntax.
     Only optionally sets the runvar, if you pass --define.  (This
      will make it easier to play around with interactively.)
     Broken the pieces of work out into subroutines for clarity.
     Comment about guest storage slightly edited and rewrapped.
     Guest storage runvars and perl variable names etc. renamed to
      `gueststorage' rather than `guest_storage'.
     LVM and VG names and perl variable names changed to be clearer.
     Install `ed' too.
     Use lv_create and toolstack()->block_attach.
     Dropped ack from Ian Campbell.

9 years agosg-run-job: Provide infrastructure for layers of nesting
Ian Jackson [Tue, 30 Jun 2015 16:19:32 +0000 (17:19 +0100)]
sg-run-job: Provide infrastructure for layers of nesting

Provides nested-layer-descend, which can be called in an individual
test job at the appropriate point (after the L1 has been set up).

The inner host is a guest of the outer host; powering it off means
destroying it.  Putting the poweroff at this point in the loop, rather
than in per-host-finish, avoids powering off physical servers.  The
use of `.'  rather than `!.' for iffail means we do not power off
after failures (as we might want to preserve the state for debugging
etc).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v14: Squash syntax fix from Robert Ho into this patch
v15: Remove spurious "=" from final-poweroff step invocation

9 years agosg-run-job: Break out per-host-prep and per-host-finish
Ian Jackson [Sat, 31 Oct 2015 03:31:54 +0000 (11:31 +0800)]
sg-run-job: Break out per-host-prep and per-host-finish

No functional change.

We now call the per-host-ts finish steps unconditionally, rather than
only if !$need_build_host, per-host-ts is (complicated) no-op if
$need_build_host, since in that case $need_xen_hosts is {}.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Squash typo fix from Robert into this patch

9 years agoToolstack::xl: Provide block_attach method
Ian Jackson [Fri, 25 Sep 2015 18:03:06 +0000 (19:03 +0100)]
Toolstack::xl: Provide block_attach method

It is possible that this may work some of the time with xm, so I have
taken no measures to prevent it running then.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Signed-off-by: Robert Hu <robert.hu@intel.com>
Tested-by: Robert Hu <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
----
v14: New patch
v15: Fix missing $ho assignment in block-attach method

9 years agoLVM: Break out lv_create
Ian Jackson [Fri, 25 Sep 2015 18:01:46 +0000 (19:01 +0100)]
LVM: Break out lv_create

We are going to want to reuse this.

lv_create doesn't (want to) take a $gho, but the $vg and $lv names
directly (so that callers can use it when they don't have a suitable
$gho whose $gho->{Lvdev} they want to use).

In the one existing call site we pass $gho->{Vg} and $gho->{Lv} so
that the effect is the same.

There is a minor functional change: $gho->{Lvdev} has been put through
lv_dev_mapper.  But we don't care about that in lv_create (since the
LVM operations, and dd, are perfectly happy to use the `real',
non-/dev/mapper, names).  So we can just use /dev/$vg/$lv.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Signed-off-by: Robert Ho <robert.hu@intel.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Discuss change to /dev/$vg/$lv from $gho->{Lvdev}.
v14: New patch
v15: Change some trivial typo, so to resolve conflicts with
     production tree.

9 years agoawait_tcp(): Run check_ip on each loop iteration
Robert Ho [Sat, 15 Aug 2015 12:37:50 +0000 (20:37 +0800)]
await_tcp(): Run check_ip on each loop iteration

await_tcp is often invoked after a reboot.

In this situation the target's IP address may change.  If this happens
while await_tcp is running, we would continue to poll the old IP address.
Fix this by running target_check_ip on each iteration.

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Dropped change to selecthost, which was in code which is no
      longer present in this version of the series.
     Rewritten to use target_check_ip.
     Dropped IMO-unnecessary comment.

9 years agotarget_check_ip: Rename and improve from guest_check_ip
Ian Jackson [Fri, 25 Sep 2015 17:07:42 +0000 (18:07 +0100)]
target_check_ip: Rename and improve from guest_check_ip

Make this function suitable for running on targets with static IP
addresses.  (Ie, on physical hosts.)  Accordingly, rename it and
adjust all call sites.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: New patch

9 years agoDhcpWatch::leases: Fix a reporting message
Ian Jackson [Fri, 25 Sep 2015 17:08:45 +0000 (18:08 +0100)]
DhcpWatch::leases: Fix a reporting message

This talks about `guest_check_ip', but this code is now factored out
into a method.  Use the correct method name in reporting.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: New patch

9 years agoNested hosts: Provide PDU power method
Robert Ho [Fri, 28 Aug 2015 01:52:15 +0000 (09:52 +0800)]
Nested hosts: Provide PDU power method

This `guest' power method uses VM create/destroy.  It is automatically
used for nested hosts.  It would not make much sense to configure it
manually.

For nested host/guest, its power on/off method shall be
its host invoke $(toolstack)->create/destroy method.

Signed-off-by: Robert Ho <robert.hu@intel.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Mostly rewritten by iwj

9 years agoselecthost: Support nested hosts (guests which are also hosts)
Ian Jackson [Fri, 25 Sep 2015 16:24:00 +0000 (17:24 +0100)]
selecthost: Support nested hosts (guests which are also hosts)

We introduce a new syntax: instead of a hostname (which might appear
in a command line argument to a ts-* script and hence be passed to
selecthost, or which might be in a runvar), we now support
<hostspec>:<domname>.

Such `hosts' (let us refer to such a thing as an L1, although in
principle further nesting may be possible) are expected to be
dynamically created.  So they do not have flags and properties in the
configuration (or in an Executive instance's database).

The IP address is determined dynamically from the leases file.  If the
L1 is not running, then no IP address may be found.  This is not an
error.  Users of this facility will need to make sure that ts-*
scripts which are unaware of the L1's special status are only invoked
when it is known that the L1 is up and has obtained its IP address.

`Power cycling' the L1 will be done by VM control operations in the
L0; this will come in a subsequent patch.

`Serial access' to the L1 guest will likewise need to be done via the
console arrangements in L0.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: New patch

9 years agomake-flight: Break out usual_debianhvm_image and honour DEBIAN_IMAGE_VERSION
Ian Jackson [Wed, 11 Nov 2015 16:35:32 +0000 (16:35 +0000)]
make-flight: Break out usual_debianhvm_image and honour DEBIAN_IMAGE_VERSION

No functional change.  (Verified for xen-unstable with
standalone-generate-dump-flight-runvars.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: New patch.

9 years agoselecthost: Minor cleanups
Ian Jackson [Fri, 25 Sep 2015 16:19:35 +0000 (17:19 +0100)]
selecthost: Minor cleanups

Document the syntax for $ident.

Log the ident as well as the selected hostname.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Robert Ho <robert.hu@intel.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Fix typo in doc message.
v14: New patch

9 years agocs-adjust-flight: Allow adjusting "this" flight
Ian Jackson [Fri, 25 Sep 2015 16:21:16 +0000 (17:21 +0100)]
cs-adjust-flight: Allow adjusting "this" flight

This allows cs-adjust-flight to be run by hand to adjust runvars, in a
flight being used with hand-invocation of ./ts-* scripts.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Better documentation of running: flight status
v14: New patch

9 years agocs-adjust-flight: Add some missing doc comment info
Ian Jackson [Fri, 25 Sep 2015 16:21:01 +0000 (17:21 +0100)]
cs-adjust-flight: Add some missing doc comment info

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: New patch

9 years agobuild_clone: git clean newly cloned trees
Ian Jackson [Thu, 29 Oct 2015 15:51:01 +0000 (15:51 +0000)]
build_clone: git clean newly cloned trees

This may seem redundant, however:

git does not track empty directories.  So it can happen that a
directory is created as part of `git clone', but is empty in the
revision switched to with `git checkout'.

In this situation, the tree we are going to build ought not to contain
this directory, because that directory will not (in general) be
produced, eg when the revision being switched to becomes master.

We can use git clean to produce a working tree whose contents -
including the presence or absence of empty directories - depends only
on the commit we are trying to check out, and not on the previous
states of the git history or working tree.

For example, if a directory is made empty (ie, deleted, since git does
not distinguish) in xen.git#staging, osstest's clones of
xen.git#master will produce the directory, but `git checkout' of
staging won't delete it.  If the xen.git build system mistakenly
depends on this directory, we won't detect this until the deletion
reaches master.  This situation actually occurred with xen.git#598e97f
"tools/python: remove broken xl binding" (fixed in b261366f).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
9 years agocs-bisection-step: Mark which revision(s) changed in each node
Ian Jackson [Fri, 23 Oct 2015 16:12:48 +0000 (16:12 +0000)]
cs-bisection-step: Mark which revision(s) changed in each node

We compare the revision rtuple elements with each of the parents, and
mark changes by surrounding the revision id with *asterisks*.

In the SVG output, we have to strip these when generating the URL, and
we can show this by emboldening the relevant revision instead.  This
involves changing the ther output to be of normal weight by deleting
the emboldening output by dot.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agocs-bisection-step: Make hyperlinks in SVG revision graph
Ian Jackson [Fri, 23 Oct 2015 13:46:04 +0000 (13:46 +0000)]
cs-bisection-step: Make hyperlinks in SVG revision graph

Make various elements in the output into hyperlinks and hence document
that the SVG version of the graph is best to use.

AFAICT dot does not provide a way to put literal SVG elements into its
output.  So we postprocess it.  Luckily we produced the input to dot
so we know a lot about what the output will look like.

In theory it would be better to feed the SVG into an XML parser and do
this editing at the ESIS level, via XSLT.  However, I don't understand
XSLT, and this regexp-based version will work until the authors of dot
decide to change the output syntax.  If they do, the hyperlinking will
go away, but everything else will still work.  I think this approach
will be less effort overall.  This is particularly true as even using
XSLT would involve us knowing how dot's output is structured, and
changes to the syntax of the dot output are not very likely unless the
dot authors also change the semantics.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agocs-bisection-step: Generate an SVG graph too
Ian Jackson [Fri, 23 Oct 2015 13:45:22 +0000 (13:45 +0000)]
cs-bisection-step: Generate an SVG graph too

This is going to let us provide a version with hyperlinks.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
9 years agoap-push: Never push an empty revision
Ian Jackson [Wed, 21 Oct 2015 12:51:43 +0000 (13:51 +0100)]
ap-push: Never push an empty revision

ap-push should not be used to delete the target branch.

This safety catch will prevent accidental deletion of branches if
there is an undetected failure of whatever is generating the revision
to push.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix typo in commit message

9 years agoms-queuedaemon: Do not spin if client input is delayed/truncated
Ian Jackson [Tue, 20 Oct 2015 17:23:02 +0000 (18:23 +0100)]
ms-queuedaemon: Do not spin if client input is delayed/truncated

chan-read-data would spin if `read' returns early because of
nonblocking mode.

Check whether the return value is the empty string (which can only
happen on eof or nonblocking lack of data, and we checked eof just
before), and if so, simply return.  The fileevent remains set up so we
will be called again when more data arrives.

(Deployment/testing note: this change is currently live in Cambridge,
as I cowboyed it directly into ~osstest/daemons-testing.git, on
observing this misbehaviour.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoREADME.bisection: New consumer-oriented document
Ian Jackson [Mon, 19 Oct 2015 16:06:43 +0000 (17:06 +0100)]
README.bisection: New consumer-oriented document

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v2: Untabify

9 years agocs-bisection-step: Print url for last failure repro flight
Ian Jackson [Mon, 19 Oct 2015 15:58:15 +0000 (16:58 +0100)]
cs-bisection-step: Print url for last failure repro flight

Print the url of the logs for last failure repro flight.  This saves
looking up the graph and manually constructing the url.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agocri-bisect: Change bisection report intro slightly
Ian Jackson [Mon, 19 Oct 2015 15:54:50 +0000 (16:54 +0100)]
cri-bisect: Change bisection report intro slightly

Remove space from `xen branch'; this is `xenbranch' in most other
places.

Use `testid' rather than `test' for the failing step testid.  This is
what it is called elsewhere.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoRename README.bisection to NOTES.bisection
Ian Jackson [Mon, 19 Oct 2015 14:38:55 +0000 (15:38 +0100)]
Rename README.bisection to NOTES.bisection

This contains osstest-developer-oriented information about the
bisector, and is not really pitched at osstest consumers.  (It's also
rather more of a sketch than a complete algorithm doc, and somewhat
out of date.)

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agots-host-ping-check: Properly append the ping output
Ian Jackson [Mon, 19 Oct 2015 14:36:40 +0000 (15:36 +0100)]
ts-host-ping-check: Properly append the ping output

On Linux opening /dev/stderr produces a non-O_APPEND open-file, with
its own file position pointer, even if stderr was opened O_APPEND.

As a result the logfile from this step would be mangled.  So use
`tee -a'.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
9 years agoap-fetch-version-old: Fix qemu branch handling after mergin of qemu trees
Ian Campbell [Fri, 23 Oct 2015 10:00:27 +0000 (11:00 +0100)]
ap-fetch-version-old: Fix qemu branch handling after mergin of qemu trees

ap-fetch-version-old should always reference the output gate, but
99e92a6b3991 "Switch to merged qemu-xen{,-traditional}.git trees"
switched it to use TREE_QEMU_UPSTREAM directly, which can be
overridden by cr-daily-branch. This broke at least when
OSSTEST_BASELINES_ONLY=y since "cr-daily-branch qemu-mainline" ends up looking
for an "upstream-tested" branch in the qemu.org git tree, when it should be
looking at our output tree on xenbits.

Follow pattern of TREE_LINUX and set BASE_TREE_QEMU_UPSTREAM to the
output gate and then conditionally set TREE_QEMU_UPSTREAM to the
BASE_TREE if it is not already set. Switch ap-fetch-version-old to use
BASE_TREE.

I have confirmed that for
qemu-{mainline,upstream-unstable,4.6-testing} both
 TREE_QEMU_UPSTREAM=git://git.qemu.org/qemu.git OSSTEST_BASELINES_ONLY=y ./ap-fetch-version-old $branch
and
 TREE_QEMU_UPSTREAM=git://git.qemu.org/qemu.git ./ap-fetch-version-old $branch
are consulting the correct trees (and produce the same answers) and that

./ap-fetch-version $branch is also correct in each case.

I have done a dummy cr-daily-branch qemu-mainline (with standalone make-flight)
with baselines forced and it now appears correct.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>