Ian Jackson [Thu, 10 Dec 2015 13:26:00 +0000 (13:26 +0000)]
Schema: Support database schema updates
See schema/README.schema, introduced in this patch, for the design.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v4: Add comment about test db safety catch.
v3: Fix spurious message from ./mg-schema-updates apply.
Fix grammar error in README.updates.
v2: Slight increase schema update name length format.
Docs fixes:
Change erroneous `three' to `four'.
Change `state' to `status' throghout.
Explain scope of <status>.
Sort out (and renumber) `Update order for Populate-then-rely'.
Sort out "Statuses" explanations.
Encourage use of DML update, rather than ad-hoc scripts,
for populating new columns.
Ian Jackson [Thu, 10 Dec 2015 12:13:58 +0000 (12:13 +0000)]
Schema: Remove SET OWNER and GRANT/REVOKE from schema/initial.sql
Really, we don't want the initial schema setup to mess about with
permissions. Instead, we simply expect to run the creation as the
correct role user.
So:
- Remove the code in mg-schema-test-database to remove the
permission settings from initial.sql;
- Instead, run exactly that code on initial.sql and commit the
result.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 11 Dec 2015 16:13:00 +0000 (16:13 +0000)]
Executive DB: Reduce strength of DB locks
The purpose of these locks is partly to prevent transactions being
aborted (which I'm not sure the existing code would in practice cope
with, although this is a bug) and also to avoid bugs due to the fact
that
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
does not mean that the transactions are necessarily serialisable!
http://www.postgresql.org/docs/8.3/static/transaction-iso.html
In SQL in general it is possible for read-only transactions to
conflict with writing transactions.
However, in PostgreSQL this is not a problem because Postgres uses
multi-version concurrency control: it retains the old version of the
data while the read transaction is open:
http://www.postgresql.org/docs/8.3/static/transaction-iso.html
So a read transaction cannot cause a write transaction to abort, nor
vice versa. So there is no need to have the database explicit table
locks prevent concurrent read access.
Preventing concurrent read access means that simple and urgent updates
can be unnecessarily delayed by long-running reader transactions in
the history reporters and archaeologists.
So, reduce the lock mode from ACCESS EXCLUSIVE to ACCESS. This still
conflicts with all kinds of updates and prospective updates, but no
longer with SELECT:
http://www.postgresql.org/docs/8.3/static/explicit-locking.html
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix grammar and typo in commit message.
Ian Jackson [Fri, 11 Dec 2015 16:04:11 +0000 (16:04 +0000)]
Executive DB: Eliminate SQL locking for read-only transactions
Our transactions generally run with
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
(which, incidentally, does not mean that the transactions are
necessarily serialisable!)
In SQL in general it is possible for a read-only transaction to fail
and need to be retried because some writer has updated things.
However, in PostgreSQL this is not possible because Postgres uses
multi-version concurrency control: it retains the old version of the
data while the read transaction is open:
http://www.postgresql.org/docs/8.3/static/transaction-iso.html
(And, of course, SQLite uses MVCC too, and all transactions in SQLite
are fully serialisable.)
So it is not necessary for these read-only operations to take out
locks. When they do so they can unnecessarily block other important
work for long periods of time.
With this change, we go further from the ability to support databases
other than PostgreSQL and SQLite. However, such support was very
distant anyway because of differences in SQL syntax and semantics, our
reliance in Executive mode on Postgres's command line utilities, and
so on.
We retain the db_retry framing because (a) although the retry loop is
not necessary in these cases, the transaction framing is (b) it will
make it slightly easier to reverse this decision in the future if we
ever decide to do so (c) it is less code churn.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix minor error in in commit message
If OSSTEST_TASK is not set, we construct a <refkey> from the username
and the nodename, and look for a such a static task. If OSSTEST_TASK
/is/ set would require it to contain `<taskid> <type> <refkey>'.
In this patch, permit OSSTEST_TASK to be set simply to the <refkey>.
This is much more convenient and doesn't involve manually looking up
taskids. The risk of error seems negligible.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 7 Dec 2015 16:33:57 +0000 (16:33 +0000)]
mg-schema-test-database: Safety catch in JobDB database open
When we open the `osstest' database, see whether this is a parent DB
(main DB) from which a test DB has been spawned by this user.
If it has, bomb out, unless the user has specified a suitable regexp
matching the DB name in the env var
OSSTEST_DB_USEREAL_IGNORETEST
This means that when a test database is in play, the user who created
it cannot accidentally operate on the real DB.
The safety catch does not affect Tcl programs, which get the DB config
directly, but in general that just means sg-execute-flight and
sg-run-job which already have a fair amount of safety catch because
they demand flight numbers.
mg-schema-test-database hits this feature over the head. We assume
that the caller of mg-schema-test-database knows what they are doing;
particularly, that if they create nested test DBs (!), they do not
need the assitance of this feature to stop themselves operating
mg-schema-test-database incorrectly. Anyone who creates nested test
DBs will hopefully recognise the potential for confusion!
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Fix unclarity in a comment.
Ian Jackson [Mon, 7 Dec 2015 16:37:03 +0000 (16:37 +0000)]
mg-schema-test-database: Change username for back-to-main-db xref
The `username' of the xdbref task in the test db, referring to the
main db, is changed to `PARENT' (from `<username>@<nodename>').
Currently this is purely cosmetic, but it is going to be useful to
distinguish the two cases:
* This is a test DB and contains references to a parent
* This is a parent DB (probably the main DB) which contains
references to child test DB(s).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v3: Fix `DBS' in commit message to `DB(s)'.
v2: New patch
Ian Jackson [Fri, 4 Dec 2015 18:24:44 +0000 (18:24 +0000)]
mg-schema-test-database: Sort out daemons; provide `daemons' subcommand
We arrange for the test configuration to look for the daemons on a
different host and port, and we provide a convenient way to run such a
pair of daemons.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Moved setting of *Daemon{Host,Port} to this patch (was
previously in `mg-schema-test-database: New script')
Ian Jackson [Fri, 4 Dec 2015 17:57:54 +0000 (17:57 +0000)]
mg-schema-test-database: New script
This allows a user in non-standalone mode to make a whole new test
database, which is largely a clone of the original database.
The new db refers to the same resources (hosts), and more-or-less
safely borrows some of those hosts.
Currently we don't do anything about the queue and owner daemons.
This means that queue-daemon-based resource allocation is broken when
clients are pointed at the test db. But non-queue-based allocation
(eg, ./mg-allocate without -U) works, and the test db can be used for
db-related experiments and even support individual ts-* scripts (other
than ts-hosts-allocate of course).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Do not set *Daemon{Host,Port} - move this chunk to a later patch
Ian Jackson [Wed, 25 Nov 2015 15:34:04 +0000 (15:34 +0000)]
cri-getconfig: Provide debugging for get_psql_cmd
This allows us to execute only the first <some number> SQL
invocations. The first non-executed one is dumped, instead, by having
get_psql_command print a rune involving ./mg-debug-fail (which the
caller will then execute).
The locking makes things work roughly-correctly if get_psql_cmd is run
in multiple processes at once: it is not defined exactly which
invocations get which counter values, but they will all work properly
and get exactly one counter value each.
If set -x is in force, turn it off for get_psql_cmd: our perl rune is
uninteresting to see repeated ad infinitum in debugging output.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Fri, 4 Dec 2015 18:12:38 +0000 (18:12 +0000)]
mg-debug-fail: New utility script for debugging
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Use "egrep ''" rather than "egrep .". Both sanitise
missing-final-newline but "egrep ''" will print blank lines,
which is desirable here.
Ian Jackson [Fri, 4 Dec 2015 18:00:48 +0000 (18:00 +0000)]
Configuration: No longer set password=<~/.xen-osstest/db-password>
Instead, expect the user to provide ~/.pgpass.
This is a good idea because we don't really want to be handling
passwords ourselves if we can help it. And, we are shortly going to
want to do some exciting mangling of the database access
configuration, which would be complicated by the presence of this
password expansion.
This may break for some users of existing Executive (non-standalone)
setups which are using production-config-cambridge or the default
built-in configuration.
DEPLOYMENT NOTE: After this passes the push gate in Cambridge,
/export/home/osstest/.{xen-,}osstest/db-password should be deleted to
avoid confusion in the future.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Thu, 3 Dec 2015 14:55:48 +0000 (14:55 +0000)]
cr-try-bisect-adhoc: Set OSSTEST_PRIORITY=-30
This makes adhoc bisects slightly more important than smoke tests, on
the basis that a smoke test can choose another host while an adhoc
bisect cannot.
Document this is README.planner and while there make a note of the
usage of OSSTEST_RESOURCE_WAITSTART by cr-try-bisect.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 4 Dec 2015 13:57:59 +0000 (13:57 +0000)]
production-config-cambridge: Use new squid proxy
Specify both HttpProxy and DebianMirrorProxy. In my tests this seems
to improve some of the apparently-intercepting-proxy-related failures,
and it will certainly improve logging.
I set DebianMirrorProxy too so that queries to security.d.o go through
the proxy. Ideally we would have a apt cache that could be used as an
http proxy rather than as an origin server; when that happens we can
set DebianMirrorProxy to point to it and do away with DebianMirrorHost
(as we do in Massachusetts).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Wed, 2 Dec 2015 16:05:04 +0000 (16:05 +0000)]
cr-try-bisect-adhoc: Set laundered_testid so graph URL is correct
Otherwise the testid is missing from the filename, resulting in e.g.
http://osstest.test-lab.xenproject.org/~osstest/pub/results-adhoc/bisect/xen-unstable/test-amd64-amd64-qemuu-nested-intel..svg
Instead of test-amd64-amd64-qemuu-nested-intel.debian-hvm-install-l1-l2.svg
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 27 Nov 2015 16:36:23 +0000 (16:36 +0000)]
sg-run-job: Coalesce a couple of repetitions
Fold `guest-localmigrate.2' into `guest-localmigrate/x10' and move
`guest-start.2' to after `guest-start.repeat' (reversing the contents
of the latter so that the start comes before the stop).
(guest-start.2 is still necessary because the start/stop test leaves
the guest stopped, whereas the subsequent destroy test ought happen
with the guest running.)
This change will allow the heisenbug compensator to see more of these
failures as the same failures.
The overall effect includes a reduction of the number of localhost
migrations from 11 to 10, but this is better than leaving a misleading
testid containing the string `x10' (or changing the testid).
It is best to fold this way, keeping the testid of the step which
previously had most of the regressions, because: the alternative,
keeping the testid of the low-repetition step, would allow osstest to
use previous lucky passes of the low-repetition step to justify
current failures of the now-high-repetition step.
To check that the effect of the patch is as intended, I ran a before
and after run with OSSTEST_SIMULATE=1, and (a) collected and sedded
and diffed the sg-run-job transcripts and (b) looked in the db.
I also ran a real test (65261 in the Xen Project test lab) with a very
similar version, which passed, and will re-run that before pushing.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Do not increment count of migration tests so as to make
testid misleading.
Do the change to the start/stop test differently.
Ian Jackson [Mon, 30 Nov 2015 13:35:54 +0000 (13:35 +0000)]
cs-bisection-step: Limit size of revision log included in reports
There is a limit in cr-daily-branch, but none in cs-bisection-step.
adhoc-revtuple-generator could usefully have this built in but that's
not so simple, so do it again here. We already slurp the whole thing
into core so from a resource usage point of view we might as well do
the length check here too.
Reported-by: David Vrabel <david.vrabel@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix typo in message.
Ian Jackson [Fri, 27 Nov 2015 11:36:05 +0000 (11:36 +0000)]
README.email: Document `fail in 58948 REGR. vs. 63449'
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Change an example number to improve clarity.
Ian Jackson [Wed, 25 Nov 2015 11:38:39 +0000 (11:38 +0000)]
Executive HTML output: More varieties of grey fruit, fewer bananas
The use of yellow for `preparing' and `running' is particularly
inappropriate in the projection summary, but is also rather misleading
when showing cancelled flights.
Use shades of grey for the different levels of in-progress-ness.
Darker shades are `more running', which seems to align better with the
shades in `Scheduled' in the flight summary.
Yellow now definitely means something is `broken', or worse.
The two places where this needs to be changed are actually
meaningfully different: report_run_getinfo works on job statuses,
whereas sg-report-flight handles only steps, which have many fewer
statuses.
Here are some samples of the ouput, from the Citrix Cambridge
instance:
Ian Campbell [Mon, 30 Nov 2015 11:58:42 +0000 (11:58 +0000)]
ts-debian-di-install: Don't set runvars for netboot kernel+ramdisk as outputs
Currently these runvars are either URLs provided by the definition
(e.g. make-flight) or output controller-relative paths created by the
execution (in the case where they aren't from the definition).
This wierd dual-semantics is confusing and wrong, and in particular is
broken if the test step is rerun (e.g. in standalone mode).
In the case where they are outputs only these paths is information
only. The information is already available in the full logs so
dropping the runvars here merely removes the information from the
summary table. It's not so useful that this is an issue.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 17 Nov 2015 17:13:46 +0000 (17:13 +0000)]
make-flight: move in-lined branch vs arch filtering into callbacks
No change to the output of standalone-generate-dump-flight-runvars
The inlined xenbranch vs arch filters remain where they are since they
are common (in that they reflect the addition and removal of arches)
and apply equally to all make-*-flight.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 17 Nov 2015 17:13:45 +0000 (17:13 +0000)]
make-flight: consolidate branch_filter_callback for builds and tests
Currently we have a test_matrix_branch_filter_callback which filters
jobs based on the $xenarch and $branch and a separate more adhoc
filter inline for the build jobs.
This has lead to things getting out of sync in the past (e.g. recently
we dropped armhf tests from the linux-3.10 and -3.14 branches but not
the build jobs).
Add a new build_matrix_branch_filter_callback and for make-flight
cause this and test_matrix_branch_filter_callback to use a common
helper.
The adhoc filtering in the build loop remains and will be tidied up
next.
For make-distros-flight just add a nop build filter alongside the test
filter.
No change to the output of standalone-generate-dump-flight-runvars.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 17 Nov 2015 17:13:44 +0000 (17:13 +0000)]
make-*flight: Reorder.
I just got tripped up again by putting a build job filter definition
after the call to create_build_jobs. Reorder the make-*flight scripts
to reduce the probability of me doing so any more times.
The general order of these scripts is now:
- job filter callbacks
- test job creation
- top-level code which drives the process.
No change to the output of standalone-generate-dump-flight-runvars.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 17 Nov 2015 17:13:43 +0000 (17:13 +0000)]
mfi-common: Drop armhf build jobs on older linux-* branches
test_matrix_branch_filter_callback was recently updated to exclude
testing of linux-3.10 (54f237784d4b) and 3.14 (b0c5663a03e7), however
this only excludes test and not build jobs so we were still trying to
build the arm kernel there.
This changes the build job filtering to be in sync with the test job
filtering, the net result is to remove build-armhf* from linux-3.10
and linux-3.14.
A subsequent patch will try and combine those two filters into one to
prevent this skew happening again.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Tue, 24 Nov 2015 15:39:00 +0000 (15:39 +0000)]
Debian: Move sysvinit-core install from preseed_create to preseed_base
preseed_create is used only for physical host installs, but we want
this workaround to apply to at least ts-debian-hvm-install'd guests
which are going to be used for nested testing.
I can't see any harm in doing this globally for all Debian HVM guests,
as well as PV guests installed using Debian Installer.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Campbell [Mon, 23 Nov 2015 11:54:11 +0000 (11:54 +0000)]
Set {ident}_suite runvar when install a Debian guest.
Currently those places which want this open code a lookup of the
{ident}_suite runvar with a fallback to the configuration file.
However selecthost was missing such a lookup in the case where it is
constructing a nested L1 host (which begins from the selectguest
template), which lead to ts-xen-install on Jessie missing the
installation of libnl-route-3-200.
Fix this by providing debian_guest_suite($gho) which as well as
initialising $gho->{Suite} stores an {ident}_suite runvar (taking care
to handle the case where one is already set by e.g. make-flight). For
convenience debian_guest_suite() also returns the suite name.
ts-debian-install, ts-debian-di-install and ts-debian-hvm-install now
use debian_guest_suite instead of open coding the lookup.
The final piece of the puzzle is to have selectguest() pickup the
{ident}_suite runvar (if it is set) and initialise $gho->{Suite} from
it.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 23 Nov 2015 15:53:53 +0000 (15:53 +0000)]
sg-run-job: Make nested-layer-descend insist on $ok
per-host-ts does nothing if !$ok, so per-host-prep's individual steps
become no-ops if the host prep fails, which prevents blundering on.
But per-host-prep does not throw.
The other call site explicitly avoids calling the recipe script if
!$ok. nested-layer-descend is for calling from within a recipe, so we
need to throw an exception to abort the script, if !$ok.
Reported-by: Ian Campbell <ian.campbell@citrix.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Campbell [Tue, 17 Nov 2015 10:58:39 +0000 (10:58 +0000)]
make-flight: Stop testing ARM for linux-3.14 flights
Since d4dba6183d61 "ts-kernel-build: Include dtbs in dist file" we
know require a "make dtbs_install" target, which was only added after
3.14.
None of the ARM h/w in the XenProject test colo can run with a kernel
this old. We do have a system in the Citrix instance in Cambridge
which does but the complexity vs benefit of testing this doesn't
warrant continuing to test 3.14.
Our current baseline for other ARM testing is a 3.16 based
linux-arm-xen branch.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Campbell [Wed, 14 Oct 2015 09:45:36 +0000 (10:45 +0100)]
standalone: only rotate logs if savelog is available
`savelog' comes from the `debianutils' package and so is unlikely to
be available elsewhere. Revert to the old behaviour of clobbering the
logs in this case.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 16 Nov 2015 15:21:12 +0000 (15:21 +0000)]
logm: Introduce $logm_prefix, and annotate nested guest messages
This allows code elsewhere to annotate log messages which might
otherwise be confusing. The variable should be localised, and the
value should always start with a space, if not empty.
Use this to annotate the calls to selecthost and selectguest from
within selecthost-for-an-L1. Otherwise some of the log messages can
be very confusing.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 16 Nov 2015 12:33:51 +0000 (12:33 +0000)]
Log capture pathnames: Include host name (and nested L0 name)
Rather than using $gho->{Name} (and, sometimes, separately
$ho->{Name}), use hostnamepath().
This means that the hostname is always included in the standard way.
The filename structure for xenctx and console snapshots changes
slightly.
In two cases this means that the hostname is now included where it
wasn't previously. That is helpful if due to some insanity the same
guest is present on more than one host.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 16 Nov 2015 12:26:11 +0000 (12:26 +0000)]
Nested hosts: Print L0 as well as L1 parent name in selectguest
Now that we have hostnamepath_list, we can remove the ad-hoc
expansion "$gn on $...{Name}" with a recipe which ascends through the
applicable nesting levels.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Tue, 17 Nov 2015 11:48:50 +0000 (11:48 +0000)]
ms-flights-summary: Improve the overview table
- Align the queued/preparing/running/etc. totals into table columns.
- Use <strong> to highlight `queued' jobs.
- Use <strong> to highlight the end time if it is the flight end,
not just a phase end.
- No longer print the `total unqueued' separately.
- Show the `common info'.
Replace much of the HTML generation with plain literal or
almost-literal HTML, since the HTML is complex to generate but easy to
write.
An example of the output can be seen here:
http://xenbits.xen.org/people/iwj/2015/flights-summary.html
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 16 Nov 2015 12:19:54 +0000 (12:19 +0000)]
Nested hosts: Use hostnamepath() in create_webfile
create_webfile needs a pathname in the shared public-html directory.
These paths need to be (a) stable (b) unique across all running jobs.
We achieve this by basing the filenames on the hostname and (for a
guest) the guest name.
But for an L2 guest we need to include the physical host name too,
because the L1 `host' is not unique.
Fix this by using hostnamepath(), replacing the open-coded single
iteration.
Reported-by: Ian Campbell <ian.campbell@citrix.com> CC: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Wed, 11 Nov 2015 11:51:37 +0000 (11:51 +0000)]
Serial::xenuse: Send xenuse output to /dev/null
Like sympathy, attaching via xenuse causes xenuse to send output from
the host to its own stdout.
But we don't want the ts-logs-capture stdout to contain this serial
output, interleaved with its own log messages. We'll capture the
whole serial log from the xenuse logfile. So redirect it to /dev/null.
I have checked that xenuse does (at least sometimes, eg when given a
nonexistent hostname) use stderr when something actually goes wrong.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: New patch
Robert Ho [Mon, 2 Nov 2015 09:21:13 +0000 (17:21 +0800)]
Serial: Add new serial method object for `guest' type
L1 guests' serial ports are owned by qemu in L0. We can send them
debug keys by writing to the qemu pipe.
(xl debug-key looks like it would be useful but it actually sends
debug keys to the hypervisor of the host it is running on. We want to
send the debug keys to the hypervisor and kernel from the outside.)
Log fetching is not needed because from the POV of the L0 the L1 is a
guest, so the L0's log capture will already fetch the L1's serial
console output.
Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Get FIFO path right.
Log the ssh command.
Put quotes around the redirection, as required.
Export sshuho from TestSupport.
Make ::guest->fetch_logs actually use @_ so that it works.
v16: Mostly rewritten: now uses new keys_real base class, and
uses the qemu pipe rather than xl debug-keys.
v15: New patch.
Ian Jackson [Tue, 10 Nov 2015 19:11:01 +0000 (19:11 +0000)]
Serial: Factor out Osstest::Serial::keys_real
The sympathy and xenuse serial modules had too much in common. Factor
out the common code, which is now responsible for
- knowledge of the Xen console switch
- splitting strings up into individual keys
- timing decisions
- error trapping and logging
This new class is an abstract base class for the concrete serial
method classes, and calls back to its derived class to prepare, send
each actual key, and shut down.
There is some functional change: notably, after failure to send the
first debug key, sending the remainder will not be attempted.
While we're here, fix a typo `dettach' to `detach'.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Move `force attach' and `force detach' writes into xenuse.pm
where they belong.
Add typo fix.
Ian Jackson [Tue, 10 Nov 2015 18:00:03 +0000 (18:00 +0000)]
HVM guests: Use qemu "pipe:" for serial output logging
Modern qemu has the "pipe:/PATH" character driver. This opens
/PATH.in for reading and /PATH.out for writing. In my tests, I found
that:
- contrary to the documentation, they do not need to be pipes
(at least, /PATH.out can be a file)
- but they must both already exist
- qemu will follow symlinks, so /PATH.out can be a symlink to
a file
- if /PATH.in is a fifo, qemu will tolerate other processes opening
it for writing, and writing things, only occasionally. (Probably,
qemu opens it O_RDWR; or perhaps it reopens it after EOF.)
Use this feature to achieve the following:
- guest serial output ends up in /var/log/xen/osstest-serial-GUEST.log
(which is already captured by ts-logs-capture) rather than
interleaved with qemu's stderr output (in the libxl-created logfile)
- guest serial input comes from a pipe in /root which we can open
and write to if we want to talk to the guest
We are mostly interested in the final bullet point, because that will
allow us to send debug keys to the emulated serial port of an L1
nested HVM guest.
Looking at the source code of qemu in 4.2 and 4.6 I think the above
approach will work with all relevant qemu-xen's.
If the device model version is qemu-xen-traditional, `pipe:' is not
supported. If device_model_version is not set, we will be using
whatever the xen.git we used defaults to. For Xen 4.1 and earlier
that is qemu-xen-traditional, and I'm slightly loathe to break osstest
for those earlier versions. There doesn't seem to be anything else in
the runvars that would clue us in. So be cautious and do not use the
new feature unless device_model_version is explicitly set.
The nested tests are all -qemuu so set device_model_version.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Fix fifo mode to be 600, not 700
v16: New patch
Robert Ho [Sat, 31 Oct 2015 05:42:03 +0000 (13:42 +0800)]
Osstest/Testsupport.pm: use get_target_property() for some host setup
For nested cases, nested host can inherit its host's property for
dhcp watch setup and ether_prefix property setup.
Signed-off-by: Robert Ho <robert.hu@intel.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v15: New patch
Ian Jackson [Sat, 31 Oct 2015 05:35:26 +0000 (13:35 +0800)]
Osstest/Testsupport.pm: change target's default kernkind to 'pvops'
This is safe only if no existing flights would be affected. (That is,
the meaning of no existing sets of runvars would be changed.)
To check whether this would make any difference I did some database
searches. Since any time target_kernkind_check is called it sets a
corresponding `console' runvar, I can search for `console' without a
corresponding `kernkind'. I ran this query:
select * from (select *, (select name from runvars r2 where
r2.flight=r1.flight and r2.job=r1.job and r2.name=
replace(r1.name,'console','kernkind')) kk from runvars r1 where
r1.name like '%console') iq where kk is null order by flight desc;
and it found nothing since flight 7682. So I think we can change the
default.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Signed-off-by: Robert Ho <robert.hu@intel.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v15: New patch
Ian Jackson [Thu, 10 Sep 2015 17:09:16 +0000 (18:09 +0100)]
ts-xen-install: networking: Rename `nodhcp' to `ensurebridge'
This function does not (now) always undo the DHCP configuration.
Sometimes it leaves it. Its main function is to ensure that we have
a bridge for use by guests.
So rename the function.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: This patch was previously 4/4 of a miniature series containing
a different way of dealing with the Nested HVM L1 DHCP problem.
Robert Ho [Fri, 14 Aug 2015 03:55:50 +0000 (11:55 +0800)]
ts-xen-install: Properly handle hosts without a static IP address
Check IpStatic, and if it is not set, provide a dhcp stanza in
/etc/network/interfaces, rather than an `inet static' one.
This is necessary for L1 nested hosts, because they don't have a
static IP address.
In principle this makes matters more correct for physical hosts
without static IP addresses, but these are currently not supported
by selecthost().
Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Only use `inet dhcp' if !$ho->{IpStatic}.
Robert Ho [Mon, 17 Aug 2015 08:40:22 +0000 (16:40 +0800)]
Nested HVM: Add test job to appropriate flights
Signed-off-by: longtao.pang <longtaox.pang@intel.com> Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com>
---
v17: Use usual_debianhvm_image
v14: Use default gueststorage_size, rather than setting runvar.
Dropped acked from Ian Campbell.
Robert Ho [Mon, 17 Aug 2015 09:07:02 +0000 (17:07 +0800)]
Nested HVM: Provide test-nested recipe
Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: ts-nested-setup command line syntax updated.
v15: (Robert Ho) remove the unnecessary l1 destroy; as it will
implicitly powered off by framework as a nested host.
This change hasn't been confirmed by Ian Jackson yet; I
plan to separate it as fix patch but by mistake squashed
it in. Ian Jackson may want to revert this if he dosn't
agree.
Robert Ho [Mon, 17 Aug 2015 07:58:01 +0000 (15:58 +0800)]
Nested HVM: Provide ts-nested-setup to help make L1 usable as a host
* Provide the L1 with some storage for its own guests' disks
* Install some packages in the L1
* Optionally, set a runvar defining the L1 for the rest of the job
The recipe is going to run ts-xen-install etc.
Signed-off-by: longtao.pang <longtaox.pang@intel.com> Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Use target_check_ip (renamed, earlier, in a new patch)
ts-nested-setup now has a default gueststorage_size of 20G
and this is implicitly used by make-flight.
Adjusted for new selecthost nested host syntax and correspondingly
completely changed invocation syntax.
Only optionally sets the runvar, if you pass --define. (This
will make it easier to play around with interactively.)
Broken the pieces of work out into subroutines for clarity.
Comment about guest storage slightly edited and rewrapped.
Guest storage runvars and perl variable names etc. renamed to
`gueststorage' rather than `guest_storage'.
LVM and VG names and perl variable names changed to be clearer.
Install `ed' too.
Use lv_create and toolstack()->block_attach.
Dropped ack from Ian Campbell.
Ian Jackson [Tue, 30 Jun 2015 16:19:32 +0000 (17:19 +0100)]
sg-run-job: Provide infrastructure for layers of nesting
Provides nested-layer-descend, which can be called in an individual
test job at the appropriate point (after the L1 has been set up).
The inner host is a guest of the outer host; powering it off means
destroying it. Putting the poweroff at this point in the loop, rather
than in per-host-finish, avoids powering off physical servers. The
use of `.' rather than `!.' for iffail means we do not power off
after failures (as we might want to preserve the state for debugging
etc).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v14: Squash syntax fix from Robert Ho into this patch
v15: Remove spurious "=" from final-poweroff step invocation
Ian Jackson [Sat, 31 Oct 2015 03:31:54 +0000 (11:31 +0800)]
sg-run-job: Break out per-host-prep and per-host-finish
No functional change.
We now call the per-host-ts finish steps unconditionally, rather than
only if !$need_build_host, per-host-ts is (complicated) no-op if
$need_build_host, since in that case $need_xen_hosts is {}.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Tested by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Squash typo fix from Robert into this patch
Ian Jackson [Fri, 25 Sep 2015 18:03:06 +0000 (19:03 +0100)]
Toolstack::xl: Provide block_attach method
It is possible that this may work some of the time with xm, so I have
taken no measures to prevent it running then.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Signed-off-by: Robert Hu <robert.hu@intel.com> Tested-by: Robert Hu <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
----
v14: New patch
v15: Fix missing $ho assignment in block-attach method
Ian Jackson [Fri, 25 Sep 2015 18:01:46 +0000 (19:01 +0100)]
LVM: Break out lv_create
We are going to want to reuse this.
lv_create doesn't (want to) take a $gho, but the $vg and $lv names
directly (so that callers can use it when they don't have a suitable
$gho whose $gho->{Lvdev} they want to use).
In the one existing call site we pass $gho->{Vg} and $gho->{Lv} so
that the effect is the same.
There is a minor functional change: $gho->{Lvdev} has been put through
lv_dev_mapper. But we don't care about that in lv_create (since the
LVM operations, and dd, are perfectly happy to use the `real',
non-/dev/mapper, names). So we can just use /dev/$vg/$lv.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Signed-off-by: Robert Ho <robert.hu@intel.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Discuss change to /dev/$vg/$lv from $gho->{Lvdev}.
v14: New patch
v15: Change some trivial typo, so to resolve conflicts with
production tree.
Robert Ho [Sat, 15 Aug 2015 12:37:50 +0000 (20:37 +0800)]
await_tcp(): Run check_ip on each loop iteration
await_tcp is often invoked after a reboot.
In this situation the target's IP address may change. If this happens
while await_tcp is running, we would continue to poll the old IP address.
Fix this by running target_check_ip on each iteration.
Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Dropped change to selecthost, which was in code which is no
longer present in this version of the series.
Rewritten to use target_check_ip.
Dropped IMO-unnecessary comment.
Ian Jackson [Fri, 25 Sep 2015 17:07:42 +0000 (18:07 +0100)]
target_check_ip: Rename and improve from guest_check_ip
Make this function suitable for running on targets with static IP
addresses. (Ie, on physical hosts.) Accordingly, rename it and
adjust all call sites.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: New patch
Ian Jackson [Fri, 25 Sep 2015 17:08:45 +0000 (18:08 +0100)]
DhcpWatch::leases: Fix a reporting message
This talks about `guest_check_ip', but this code is now factored out
into a method. Use the correct method name in reporting.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: New patch
Robert Ho [Fri, 28 Aug 2015 01:52:15 +0000 (09:52 +0800)]
Nested hosts: Provide PDU power method
This `guest' power method uses VM create/destroy. It is automatically
used for nested hosts. It would not make much sense to configure it
manually.
For nested host/guest, its power on/off method shall be
its host invoke $(toolstack)->create/destroy method.
Signed-off-by: Robert Ho <robert.hu@intel.com> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: Mostly rewritten by iwj
Ian Jackson [Fri, 25 Sep 2015 16:24:00 +0000 (17:24 +0100)]
selecthost: Support nested hosts (guests which are also hosts)
We introduce a new syntax: instead of a hostname (which might appear
in a command line argument to a ts-* script and hence be passed to
selecthost, or which might be in a runvar), we now support
<hostspec>:<domname>.
Such `hosts' (let us refer to such a thing as an L1, although in
principle further nesting may be possible) are expected to be
dynamically created. So they do not have flags and properties in the
configuration (or in an Executive instance's database).
The IP address is determined dynamically from the leases file. If the
L1 is not running, then no IP address may be found. This is not an
error. Users of this facility will need to make sure that ts-*
scripts which are unaware of the L1's special status are only invoked
when it is known that the L1 is up and has obtained its IP address.
`Power cycling' the L1 will be done by VM control operations in the
L0; this will come in a subsequent patch.
`Serial access' to the L1 guest will likewise need to be done via the
console arrangements in L0.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v14: New patch
Ian Jackson [Wed, 11 Nov 2015 16:35:32 +0000 (16:35 +0000)]
make-flight: Break out usual_debianhvm_image and honour DEBIAN_IMAGE_VERSION
No functional change. (Verified for xen-unstable with
standalone-generate-dump-flight-runvars.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: New patch.
Ian Jackson [Fri, 25 Sep 2015 16:19:35 +0000 (17:19 +0100)]
selecthost: Minor cleanups
Document the syntax for $ident.
Log the ident as well as the selected hostname.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Robert Ho <robert.hu@intel.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Fix typo in doc message.
v14: New patch
Ian Jackson [Fri, 25 Sep 2015 16:21:16 +0000 (17:21 +0100)]
cs-adjust-flight: Allow adjusting "this" flight
This allows cs-adjust-flight to be run by hand to adjust runvars, in a
flight being used with hand-invocation of ./ts-* scripts.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Tested-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v17: Better documentation of running: flight status
v14: New patch
Ian Jackson [Thu, 29 Oct 2015 15:51:01 +0000 (15:51 +0000)]
build_clone: git clean newly cloned trees
This may seem redundant, however:
git does not track empty directories. So it can happen that a
directory is created as part of `git clone', but is empty in the
revision switched to with `git checkout'.
In this situation, the tree we are going to build ought not to contain
this directory, because that directory will not (in general) be
produced, eg when the revision being switched to becomes master.
We can use git clean to produce a working tree whose contents -
including the presence or absence of empty directories - depends only
on the commit we are trying to check out, and not on the previous
states of the git history or working tree.
For example, if a directory is made empty (ie, deleted, since git does
not distinguish) in xen.git#staging, osstest's clones of
xen.git#master will produce the directory, but `git checkout' of
staging won't delete it. If the xen.git build system mistakenly
depends on this directory, we won't detect this until the deletion
reaches master. This situation actually occurred with xen.git#598e97f
"tools/python: remove broken xl binding" (fixed in b261366f).
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Andrew Cooper <andrew.cooper3@citrix.com> Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Ian Jackson [Fri, 23 Oct 2015 16:12:48 +0000 (16:12 +0000)]
cs-bisection-step: Mark which revision(s) changed in each node
We compare the revision rtuple elements with each of the parents, and
mark changes by surrounding the revision id with *asterisks*.
In the SVG output, we have to strip these when generating the URL, and
we can show this by emboldening the relevant revision instead. This
involves changing the ther output to be of normal weight by deleting
the emboldening output by dot.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 23 Oct 2015 13:46:04 +0000 (13:46 +0000)]
cs-bisection-step: Make hyperlinks in SVG revision graph
Make various elements in the output into hyperlinks and hence document
that the SVG version of the graph is best to use.
AFAICT dot does not provide a way to put literal SVG elements into its
output. So we postprocess it. Luckily we produced the input to dot
so we know a lot about what the output will look like.
In theory it would be better to feed the SVG into an XML parser and do
this editing at the ESIS level, via XSLT. However, I don't understand
XSLT, and this regexp-based version will work until the authors of dot
decide to change the output syntax. If they do, the hyperlinking will
go away, but everything else will still work. I think this approach
will be less effort overall. This is particularly true as even using
XSLT would involve us knowing how dot's output is structured, and
changes to the syntax of the dot output are not very likely unless the
dot authors also change the semantics.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 20 Oct 2015 17:23:02 +0000 (18:23 +0100)]
ms-queuedaemon: Do not spin if client input is delayed/truncated
chan-read-data would spin if `read' returns early because of
nonblocking mode.
Check whether the return value is the empty string (which can only
happen on eof or nonblocking lack of data, and we checked eof just
before), and if so, simply return. The fileevent remains set up so we
will be called again when more data arrives.
(Deployment/testing note: this change is currently live in Cambridge,
as I cowboyed it directly into ~osstest/daemons-testing.git, on
observing this misbehaviour.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Jackson [Mon, 19 Oct 2015 16:06:43 +0000 (17:06 +0100)]
README.bisection: New consumer-oriented document
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com> CC: Wei Liu <wei.liu2@citrix.com>
---
v2: Untabify
Ian Jackson [Mon, 19 Oct 2015 14:38:55 +0000 (15:38 +0100)]
Rename README.bisection to NOTES.bisection
This contains osstest-developer-oriented information about the
bisector, and is not really pitched at osstest consumers. (It's also
rather more of a sketch than a complete algorithm doc, and somewhat
out of date.)
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian Campbell [Fri, 23 Oct 2015 10:00:27 +0000 (11:00 +0100)]
ap-fetch-version-old: Fix qemu branch handling after mergin of qemu trees
ap-fetch-version-old should always reference the output gate, but 99e92a6b3991 "Switch to merged qemu-xen{,-traditional}.git trees"
switched it to use TREE_QEMU_UPSTREAM directly, which can be
overridden by cr-daily-branch. This broke at least when
OSSTEST_BASELINES_ONLY=y since "cr-daily-branch qemu-mainline" ends up looking
for an "upstream-tested" branch in the qemu.org git tree, when it should be
looking at our output tree on xenbits.
Follow pattern of TREE_LINUX and set BASE_TREE_QEMU_UPSTREAM to the
output gate and then conditionally set TREE_QEMU_UPSTREAM to the
BASE_TREE if it is not already set. Switch ap-fetch-version-old to use
BASE_TREE.
I have confirmed that for
qemu-{mainline,upstream-unstable,4.6-testing} both
TREE_QEMU_UPSTREAM=git://git.qemu.org/qemu.git OSSTEST_BASELINES_ONLY=y ./ap-fetch-version-old $branch
and
TREE_QEMU_UPSTREAM=git://git.qemu.org/qemu.git ./ap-fetch-version-old $branch
are consulting the correct trees (and produce the same answers) and that
./ap-fetch-version $branch is also correct in each case.
I have done a dummy cr-daily-branch qemu-mainline (with standalone make-flight)
with baselines forced and it now appears correct.
Signed-off-by: Ian Campbell <ian.campbell@citrix.com>