Ian Jackson [Wed, 5 Aug 2020 12:23:41 +0000 (13:23 +0100)]
history reporting (nfc): Refactor to generalise, cache_read_existing
* Introduce @cache_row_key_cols and cache_row_key
* Handle $html_file pathname construction and $read_existing
at the call site.
* Rename cache_read_existing.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 17 Aug 2020 17:45:43 +0000 (18:45 +0100)]
Tcl: Use tclsh8.6
This is needed to run on buster.
I have checked that tclsh8.6 and TclX works on osstest.test-lab. TclX
seems to be provided by tcl8.4 but work with tcl8.6 (at least on
buster).
Deployment note: hosts running earlier Debian (including
osstest.xs.citrite.net, the Citrix Cambridge instance), may need
OSSTEST_DAEMON_TCLSH=tclsh8.4 or similar in ~/.xen-osstest/settings.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Mon, 10 Aug 2020 15:19:16 +0000 (16:19 +0100)]
SQL: Change "... LIKE '...\_...' ..." to "... LIKE '...\\_...' ..."
Perl's "" quotes, and corresponding <<END constructs, do
\-interpolation, so remove these \ which we carefully added in e7a408dd01184df0a57ae5d9072d15225c52a99f
SQL: Change LIKE E'...\\_...' to LIKE '...\_...'
This is only not a performance problem due to query mismathes with the
available indices, because the test indices are still present in the
Massachusetts instance.
I have verified that this has the intended chanve everywhere by
1. Double-checking that eacbh of these instances is within <<END
or "" or equivalent.
2. Running git-ls-files | xargs perl -i -pe 's/\\\\_/\\_/g'
and manually examining the diff against this patch's parent.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Jim Fehlig [Thu, 23 Jul 2020 21:31:34 +0000 (22:31 +0100)]
Install libtirpc-dev for libvirt builds
The check for XDR support was changed in libvirt commit d7147b3797
to use libtirpc pkg-config instead of complicated AC_CHECK_LIB,
AC_COMPILE_IFELSE, et. al. logic. The libvirt OSSTEST has been
failing since this change hit libvirt.git master. Fix it by adding
libtirpc-dev to the list of 'extra_packages' installed for libvirt
builds.
Signed-off-by: Jim Fehlig <jfehlig@suse.com> Reviewed-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 31 Jul 2020 10:43:55 +0000 (11:43 +0100)]
duration_estimator: Clarify recentflights query a bit
The condition on r.job is more naturally thought of as a join
condition than a where condition. (This is an inner join, so the
semantics are identical.)
Also, for clarity, swap the flight and job conditions round, so that
the ON clause is a series of r.thing = otherthing.
No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Reviewed-by: George Dunlap <George.Dunlap@citrix.com>
---
v3: Fix daft syntax error.
v2: New patch.
Ian Jackson [Fri, 24 Jul 2020 18:03:40 +0000 (19:03 +0100)]
cs-bisection-step: Use db_prepare a few times instead of ->do
With $dbh_tests->do(...), we can only get a debug trace of the queries
by using DBI_TRACE which produces voluminous output. Using our own
db_prepare invokes our own debugging.
No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch.
Ian Jackson [Mon, 27 Jul 2020 11:22:51 +0000 (12:22 +0100)]
cs-bisection-step: Generalise qtxt_common_rev_ok
* Make it into a subref which takes a $table argument.
* Change the two references into function calls using the @{...} syntax
* Move the definition earlier in the file
No change to the generated query.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch.
Ian Jackson [Fri, 24 Jul 2020 15:51:28 +0000 (16:51 +0100)]
sg-report-host-history: Rerganisation: Read old logs later
Perhaps at one point something read from these logs influenced the db
query for thye flights range, but that is no longer the case and it
doesn't seem likely to need to come back.
We want to move the per-host stuff together.
No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Now that we have an index, there is a faster way for the db to do this
query: via that index. But it doesn't like to do that if be aggregate
the queries. Experimentally, doing this query separately once per
host is significantly faster.
Also, later, it will allow us to parallelise this work.
So, we undo that. (Not by reverting, though.)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Use proper \ escaping for underscores in LIKE
Ian Jackson [Tue, 21 Jul 2020 14:36:11 +0000 (15:36 +0100)]
duration_estimator: Move duration query loop into database
Stuff the two queries together: we use the firsty query as a WITH
clause. This is significantly faster, perhaps because the query
optimiser does a better job but probably just because it saves on
round trips.
No functional change.
Perf: subjectively this seemed to help when the cache was cold. Now I
have a warm cache and it doesn't seem to make much difference.
Perf: runtime of my test case now ~5-7s.
Example queries before (from the debugging output):
Query A part I:
SELECT f.flight AS flight,
j.job AS job,
f.started AS started,
j.status AS status
FROM flights f
JOIN jobs j USING (flight)
JOIN runvars r
ON f.flight=r.flight
AND r.name=?
WHERE j.job=r.job
AND f.blessing=?
AND f.branch=?
AND j.job=?
AND r.val=?
AND (j.status='pass' OR j.status='fail'
OR j.status='truncated'!)
AND f.started IS NOT NULL
AND f.started >= ?
ORDER BY f.started DESC
With bind variables:
"test-amd64-i386-xl-pvshim"
"guest-start"
Query B part I:
SELECT f.flight AS flight,
s.job AS job,
NULL as started,
NULL as status,
max(s.finished) AS max_finished
FROM steps s JOIN flights f
ON s.flight=f.flight
WHERE s.job=? AND f.blessing=? AND f.branch=?
AND s.finished IS NOT NULL
AND f.started IS NOT NULL
AND f.started >= ?
GROUP BY f.flight, s.job
ORDER BY max_finished DESC
With bind variables:
"test-armhf-armhf-libvirt"
'real'
"xen-unstable" 1594144469
Query common part II:
WITH tsteps AS
(
SELECT *
FROM steps
WHERE flight=? AND job=?
)
, tsteps2 AS
(
SELECT *
FROM tsteps
WHERE finished <=
(SELECT finished
FROM tsteps
WHERE tsteps.testid = ?)
)
SELECT (
SELECT max(finished)-min(started)
FROM tsteps2
) - (
SELECT sum(finished-started)
FROM tsteps2
WHERE step = 'ts-hosts-allocate'
)
AS duration
With bind variables from previous query, eg:
152045
"test-armhf-armhf-libvirt"
"guest-start.2"
After:
Query A (combined):
WITH f AS (
SELECT f.flight AS flight,
j.job AS job,
f.started AS started,
j.status AS status
FROM flights f
JOIN jobs j USING (flight)
JOIN runvars r
ON f.flight=r.flight
AND r.name=?
WHERE j.job=r.job
AND f.blessing=?
AND f.branch=?
AND j.job=?
AND r.val=?
AND (j.status='pass' OR j.status='fail'
OR j.status='truncated'!)
AND f.started IS NOT NULL
AND f.started >= ?
ORDER BY f.started DESC
)
SELECT flight, job, started, status,
(
WITH tsteps AS
(
SELECT *
FROM steps
WHERE flight=f.flight AND job=f.job
)
, tsteps2 AS
(
SELECT *
FROM tsteps
WHERE finished <=
(SELECT finished
FROM tsteps
WHERE tsteps.testid = ?)
)
SELECT (
SELECT max(finished)-min(started)
FROM tsteps2
) - (
SELECT sum(finished-started)
FROM tsteps2
WHERE step = 'ts-hosts-allocate'
)
AS duration
) FROM f
Query B (combined):
WITH f AS (
SELECT f.flight AS flight,
s.job AS job,
NULL as started,
NULL as status,
max(s.finished) AS max_finished
FROM steps s JOIN flights f
ON s.flight=f.flight
WHERE s.job=? AND f.blessing=? AND f.branch=?
AND s.finished IS NOT NULL
AND f.started IS NOT NULL
AND f.started >= ?
GROUP BY f.flight, s.job
ORDER BY max_finished DESC
)
SELECT flight, job, started, status,
(
WITH tsteps AS
(
SELECT *
FROM steps
WHERE flight=f.flight AND job=f.job
)
, tsteps2 AS
(
SELECT *
FROM tsteps
WHERE finished <=
(SELECT finished
FROM tsteps
WHERE tsteps.testid = ?)
)
SELECT (
SELECT max(finished)-min(started)
FROM tsteps2
) - (
SELECT sum(finished-started)
FROM tsteps2
WHERE step = 'ts-hosts-allocate'
)
AS duration
) FROM f
Diff for query A:
@@ -1,3 +1,4 @@
+ WITH f AS (
SELECT f.flight AS flight,
j.job AS job,
f.started AS started,
@@ -18,11 +19,14 @@
AND f.started >= ?
ORDER BY f.started DESC
+ )
+ SELECT flight, job, started, status,
+ (
WITH tsteps AS
(
SELECT *
FROM steps
- WHERE flight=? AND job=?
+ WHERE flight=f.flight AND job=f.job
)
, tsteps2 AS
(
@@ -42,3 +46,5 @@
WHERE step = 'ts-hosts-allocate'
)
AS duration
+
+ ) FROM f
Diff for query B:
@@ -1,3 +1,4 @@
+ WITH f AS (
SELECT f.flight AS flight,
s.job AS job,
NULL as started,
@@ -12,11 +13,14 @@
GROUP BY f.flight, s.job
ORDER BY max_finished DESC
+ )
+ SELECT flight, job, started, status,
+ (
WITH tsteps AS
(
SELECT *
FROM steps
- WHERE flight=? AND job=?
+ WHERE flight=f.flight AND job=f.job
)
, tsteps2 AS
(
@@ -36,3 +40,5 @@
WHERE step = 'ts-hosts-allocate'
)
AS duration
+
+ ) FROM f
Reviewed-by: George Dunlap <George.Dunlap@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v4: Drop max_finished from new SELECT clause.
This column is present in the query A just for the ORDER BY.
It is not used by the perl code which actually processe
these rows. Which is just as well because the query B doesn't
have this column, so with max_finished that query is broken.
Ian Jackson [Tue, 21 Jul 2020 14:32:33 +0000 (15:32 +0100)]
duration_estimator: Return job column in first query
Right now this is pointless since the Perl code doesn't need it. But
this row is going to be part of a WITH clause soon.
No functional change.
Diffs to two example queries (from the Perl DBI trace):
SELECT f.flight AS flight,
+ j.job AS job,
f.started AS started,
j.status AS status
FROM flights f
JOIN jobs j USING (flight)
JOIN runvars r
ON f.flight=r.flight
AND r.name=?
WHERE j.job=r.job
AND f.blessing=?
AND f.branch=?
AND j.job=?
AND r.val=?
AND (j.status='pass' OR j.status='fail'
OR j.status='truncated'!)
AND f.started IS NOT NULL
AND f.started >= ?
ORDER BY f.started DESC
SELECT f.flight AS flight,
+ s.job AS job,
NULL as started,
NULL as status,
max(s.finished) AS max_finished
FROM steps s JOIN flights f
ON s.flight=f.flight
WHERE s.job=? AND f.blessing=? AND f.branch=?
AND s.finished IS NOT NULL
AND f.started IS NOT NULL
AND f.started >= ?
- GROUP BY f.flight
+ GROUP BY f.flight, s.job
ORDER BY max_finished DESC
CC: George Dunlap <George.Dunlap@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 21 Jul 2020 14:22:28 +0000 (15:22 +0100)]
duration_estimator: Explicitly provide null in general host q
Our spec. says we return nulls for started and status if we don't find
a job matching the host spec.
The way this works right now is that we look up the nonexistent
entries in $refs->[0]. This is not really brilliant and is going to
be troublesome as we continue to refactor.
Provide these values explicitly. No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 20 Jul 2020 16:36:54 +0000 (17:36 +0100)]
Executive: Use index for report__find_test
After we refactor this query then we can enable the index use.
(Both of these things together in this commit because I haven't perf
tested the version with just the refactoring.)
(We have provided an index that can answer this question really
quickly if a version is specified. But the query planner couldn't see
that because it works without seeing the bind variables, so doesn't
know that the value of name is going to be suitable for this index.)
* Convert the two EXISTS subqueries into JOIN/AND with a DISTINCT
clause naming the fields on flights, so as to replicate the previous
result rows. Then do $selection field last. The subquery is a
convenient way to let this do the previous thing for all the values
of $selection (including, notably, *).
* Add the additional AND clause for r.name, which has no logical
effect given the actual values of name, enabling the query planner
to use this index.
Perf: In my test case the sg-report-flight runtime is now ~8s. I am
reasonably confident that this will not make other use cases of this
code worse.
Perf: runtime of my test case now ~11s
Example query before (from the Perl DBI trace):
SELECT *
FROM flights f
WHERE
EXISTS (
SELECT 1
FROM runvars r
WHERE name=?
AND val=?
AND r.flight=f.flight
AND ( (CASE
WHEN (r.job) LIKE 'build-%-prev' THEN 'xprev'
WHEN ((r.job) LIKE 'build-%-freebsd'
AND 'x' = 'freebsdbuildjob') THEN 'DISCARD'
ELSE ''
END)
= '')
)
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
AND (branch=?)
ORDER BY flight DESC
LIMIT 1
After:
SELECT *
FROM ( SELECT DISTINCT
flight, started, blessing, branch, intended
FROM flights f
JOIN runvars r USING (flight)
WHERE name=?
AND name LIKE 'revision\_%'
AND val=?
AND r.flight=f.flight
AND ( (CASE
WHEN (r.job) LIKE 'build-%-prev' THEN 'xprev'
WHEN ((r.job) LIKE 'build-%-freebsd'
AND 'x' = 'freebsdbuildjob') THEN 'DISCARD'
ELSE ''
END)
= '')
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
AND (branch=?)
) AS sub WHERE TRUE
ORDER BY flight DESC
LIMIT 1
@@ -1,10 +1,10 @@
SELECT *
+ FROM ( SELECT DISTINCT
+ flight, started, blessing, branch, intended
FROM flights f
- WHERE
- EXISTS (
- SELECT 1
- FROM runvars r
+ JOIN runvars r USING (flight)
WHERE name=?
+ AND name LIKE 'revision\_%'
AND val=?
AND r.flight=f.flight
AND ( (CASE
@@ -14,8 +14,8 @@
ELSE ''
END)
= '')
- )
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
AND (branch=?)
+) AS sub WHERE TRUE
ORDER BY flight DESC
LIMIT 1
Reviewed-by: George Dunlap <George.Dunlap@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Use proper \ escaping for underscores in LIKE
Ian Jackson [Mon, 20 Jul 2020 15:58:16 +0000 (16:58 +0100)]
sg-report-flight: Use WITH clause to use index for $anypassq
Perf: runtime of my test case now ~11s
Example query before (from the Perl DBI trace):
SELECT * FROM flights JOIN steps USING (flight)
WHERE (branch='xen-unstable')
AND job=? and testid=? and status='pass'
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
LIMIT 1
After:
WITH s AS
(
SELECT * FROM steps
WHERE job=? and testid=? and status='pass'
)
SELECT * FROM flights JOIN s USING (flight)
WHERE (branch='xen-unstable')
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
LIMIT 1
In both cases with bind vars:
"test-amd64-i386-xl-pvshim"
"guest-start"
Diff to the query:
- SELECT * FROM flights JOIN steps USING (flight)
+ WITH s AS
+ (
+ SELECT * FROM steps
+ WHERE job=? and testid=? and status='pass'
+ )
+ SELECT * FROM flights JOIN s USING (flight)
WHERE (branch='xen-unstable')
- AND job=? and testid=? and status='pass'
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
LIMIT 1
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Reviewed-by: George Dunlap <george.dunlap@citrix.com>
Ian Jackson [Tue, 21 Jul 2020 11:56:42 +0000 (12:56 +0100)]
sg-report-flight: Use WITH to use best index use for $flightsq
While we're here, convert this EXISTS subquery to a JOIN.
Perf: runtime of my test case now ~200-300s.
Example query before (from the Perl DBI trace):
SELECT * FROM (
SELECT DISTINCT flight, blessing
FROM flights
JOIN runvars r1 USING (flight)
WHERE (branch='xen-unstable')
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
AND EXISTS (SELECT 1
FROM jobs
WHERE jobs.flight = flights.flight
AND jobs.job = ?)
AND r1.name LIKE 'built_revision_%'
AND r1.name = ?
AND r1.val= ?
ORDER BY flight DESC
LIMIT 1000
) AS sub
ORDER BY blessing ASC, flight DESC
WITH sub AS (
SELECT DISTINCT flight, blessing
FROM flights
JOIN runvars r1 USING (flight)
WHERE (branch='xen-unstable')
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
AND r1.name LIKE 'built_revision_%'
AND r1.name = ?
AND r1.val= ?
ORDER BY flight DESC
LIMIT 1000
)
SELECT *
FROM sub
JOIN jobs USING (flight)
- SELECT * FROM (
+ WITH sub AS (
SELECT DISTINCT flight, blessing
FROM flights
JOIN runvars r1 USING (flight)
WHERE (branch='xen-unstable')
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
- AND EXISTS (SELECT 1
- FROM jobs
- WHERE jobs.flight = flights.flight
- AND jobs.job = ?)
-
AND r1.name LIKE 'built_revision_%'
AND r1.name = ?
AND r1.val= ?
ORDER BY flight DESC
LIMIT 1000
- ) AS sub
+ )
+ SELECT *
+ FROM sub
+ JOIN jobs USING (flight)
+
+ WHERE (1=1)
+ AND jobs.job = ?
+
ORDER BY blessing ASC, flight DESC
Reviewed-by: George Dunlap <George.Dunlap@citrix.com> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 17 Jul 2020 15:04:50 +0000 (16:04 +0100)]
sg-report-flight: Ask the db for flights of interest
Specifically, we narrow the initial query to flights which have at
least some job with the built_revision_foo we are looking for.
This condition is strictly broader than that implemented inside the
flight search loop, so there is no functional change.
Perf: runtime of my test case now ~300s-500s.
Example query before (from the Perl DBI trace):
SELECT * FROM (
SELECT flight, blessing FROM flights
WHERE (branch='xen-unstable')
AND EXISTS (SELECT 1
FROM jobs
WHERE jobs.flight = flights.flight
AND jobs.job = ?)
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
ORDER BY flight DESC
LIMIT 1000
) AS sub
ORDER BY blessing ASC, flight DESC
With these bind variables:
"test-armhf-armhf-libvirt"
After:
SELECT * FROM (
SELECT DISTINCT flight, blessing
FROM flights
JOIN runvars r1 USING (flight)
WHERE (branch='xen-unstable')
AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
AND EXISTS (SELECT 1
FROM jobs
WHERE jobs.flight = flights.flight
AND jobs.job = ?)
AND r1.name LIKE 'built\_revision\_%'
AND r1.name = ?
AND r1.val= ?
ORDER BY flight DESC
LIMIT 1000
) AS sub
ORDER BY blessing ASC, flight DESC
SELECT * FROM (
- SELECT flight, blessing FROM flights
+ SELECT DISTINCT flight, blessing
+ FROM flights
+ JOIN runvars r1 USING (flight)
+
WHERE (branch='xen-unstable')
+ AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
AND EXISTS (SELECT 1
FROM jobs
WHERE jobs.flight = flights.flight
AND jobs.job = ?)
- AND ( (TRUE AND flight <= 151903) AND (blessing='real') )
+ AND r1.name LIKE 'built\_revision\_%'
+ AND r1.name = ?
+ AND r1.val= ?
+
ORDER BY flight DESC
LIMIT 1000
) AS sub
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Reviewed-by: George Dunlap <George.Dunlap@citrix.com>
---
v3: Add more formal correctness argument
v2: Use proper \ escaping for underscores in LIKE
Ian Jackson [Tue, 21 Jul 2020 11:23:14 +0000 (12:23 +0100)]
sg-report-flight: Add a comment re same-flight search narrowing
In afe851ca1771e5da6395b596afa69e509dbbc278
sg-report-flight: When justifying, disregard out-of-flight build jobs
we narrowed sg-report-flight's search algorith.
An extensive justification is in the commit message. I think much of
this information belongs in-tree, so c&p it (with slight edits) here.
No code change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 31 Jul 2020 15:52:06 +0000 (16:52 +0100)]
Disable mercurial support
This is in order that we can substantially simplify forthcoming
database changes. If mercurial support were still desired, the right
thing to do would be to rework it now along the lines of this request.
But we haven't used it for some years.
It could be reenabled later, if this work were done then. (Of course
there might be other bitrot already that we don't know about.)
CC: committers@xenproject.org Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Acked-by: Andrew Cooper <Andrew.Cooper3@citrix.com>
Ian Jackson [Mon, 27 Jul 2020 11:43:27 +0000 (12:43 +0100)]
SQL: Use "LIKE" rather than "like", etc.
This is more like the rest of the style. It will also make it easier
to find instances of the mistaken LIKE syntax.
I found these with "git grep" and manually edited them. I have
checked the before-and-after result of
find * -type f | xargs perl -i~ -pe 's/\bLIKE\b/like/g'
and it has only the few expected changes to ANDs and ORs.
No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
New in v2.
Ian Jackson [Fri, 24 Jul 2020 17:43:40 +0000 (18:43 +0100)]
schema: Add index to help cs-bisection-step
cs-bisection step basis search involves looking for recent flights
that weren't broken. A flight is broken if it has broken steps.
Make an index for this to save it scanning the steps table.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 17 Jul 2020 16:04:45 +0000 (17:04 +0100)]
schema: Provide indices for sg-report-flight
These indexes allow very fast lookup of "relevant" flights eg when
trying to justify failures.
In my ad-hoc test case, these indices (along with the subsequent
changes to sg-report-flight and Executive.pm, reduce the runtime of
sg-report-flight from 2-3ks (unacceptably long!) to as little as
5-7s seconds - a speedup of about 500x.
(Getting the database snapshot may take a while first, but deploying
this code should help with that too by reducing long-running
transactions. Quoted perf timings are from snapshot acquisition.)
Without these new indexes there may be a performance change from the
query changes. I haven't benchmarked this so I am setting the schema
updates to be Preparatory/Needed (ie, "Schema first" as
schema/README.updates has it), to say that the index should be created
before the new code is deployed.
Testing: I have tested this series by creating experimental indices
"trial_..." in the actual production instance. (Transactional DDL was
very helpful with this.) I have verified with \d that schema update
instructions in this commit generate indexes which are equivalent to
the trial indices.
Deployment: AFter these schema updates are applied, the trial indices
are redundant duplicates and should be deleted.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Reviewed-by: George Dunlap <George.Dunlap@citrix.com>
---
v2: Use proper \ escaping for underscores in LIKE
Current tests use 512MB of memory for dom0, but that's too low for a
PVH dom0 on some hosts and will cause errors because memory is
ballooned out in order to obtain physical memory ranges to map foreign
pages.
Using ballooned out pages for foreign mappings also doesn't seem to
work properly with the current Linux kernel version, so increase the
memory assigned to dom0 to 1GB for PVH dom0 tests. We should see about
reverting this when using ballooned pages is fixed.
freebsd: remove freebsd- hostflags request from guest tests
Guest tests shouldn't care about the capabilities or firmware of the
underlying hosts, so drop the request of specific freebsd-<version>
hostflags for FreeBSD guest tests.
While there request the presence of the hvm hostflag since the FreeBSD
guest tests are run in HVM mode.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Roger Pau Monne [Thu, 28 May 2020 10:26:47 +0000 (11:26 +0100)]
freebsd10: refactor code to generate jobs
Slightly adjust the code to generate the freebsd10 jobs in order to
avoid exiting early from the function if the dom0 arch is different
than i386. No functional change expected, the freebsd10 jobs are still
limited to run on an i386 dom0. No runvar diff created as part of this
change.
This is a preparatory change for adding new FreeBSD 11 and 12 jobs
that will instead use an amd64 dom0.
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 16 Jun 2020 16:42:59 +0000 (17:42 +0100)]
make-flight: Test Xen 4.9 and Xen 4.10 on stretch
Xen 4.9 doesn't build on buster and won't be fixed for that.
Xen 4.10's "-prev" migration tests also fail for the same reason.
There is a (smallish) risk that Debian will break stretch before Xen
4.10 goes completely out of support from the Xen Project. If that
happens we could revert this - but then the 4.10 -prev jobs will stop
building.
Xen 4.9 is nearly out of security support, so for the 4.9 tests this
is a no-brainer.
I have checked the runvar diff. The results are to change
all_host_di_version [*]
all_host_suite
all_hostflags
debian_suite
debianhvm_suite
host_hostflags
in many jobs in
qemu-upstream-4.9-testing
qemu-upstream-4.10-testing
xen-4.9-testing
xen-4.10-testing
[*] this is not visible in standalone-generate-dump-flight-runvars
because it always just uses `current'.
This command produces no output:
diff -ub a <(perl -pe 's/stretch/buster/g' c)
(where `a' is before and `c' is after.)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Otherwise if logs doesn't exist, the code in `standalone' which is
eventually called to build each flight will try to do it - but that
code is not idempotent in the presence of other racing copies of
itself.
Rather than trusting mkdir -p there, do it here.
No change other than to this dev-debugging script.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 9 Jun 2020 15:30:48 +0000 (15:30 +0000)]
production-config: Update to current coverity tools
The 2017.07 tarball had a leading "./" but the 2019.03 one doesn't,
so we must change CoverityToolsStripComponents.
The old tools we are currently using do not work any more - they are
rejected by Coverity Scan.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Reported-by: Matthew Krasnick <Matthew.Krasnick@synopsys.com> CC: Jan Beulich <jbeulich@suse.com> CC: Andrew Cooper <Andrew.Cooper3@citrix.com> CC: Paul Durrant <xadimgnik@gmail.com>
Ian Jackson [Wed, 27 May 2020 15:42:27 +0000 (16:42 +0100)]
setupboot_grub2: Do not boot with XSM policy etc. unless xsm=1
This prevents us from passing an XSM policy file, and
`flask=enforcing', in supposedly-non-XSM tests.
These bootloader entries can appear because the Xen upstream build
ships XSM policy files by default even if XSM is disabled in the
hypervisor, causing update-grub to generate useless `XSM enabled'
entries.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 27 May 2020 15:16:52 +0000 (16:16 +0100)]
setupboot_grub2: Copy hv command line from grub to xen.cfg
This reuses all of the stuff that update-grub, etc., have put there.
In particular without this we never have flask=enforcing!
We have to do something about the ${xen_rm_opts} that appear in these
entries. In principle there might be many variable expansions, but in
practice there is only this one It applies only to x86, and this use
of chainloading to xen.efi and reading xen.cfg applies only to arm64.
And anyway we weren't putting it these things in before. So OK...
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 21 May 2020 13:50:42 +0000 (14:50 +0100)]
buster: Install own linux-libc-dev package (!)
As reported here:
https://patchew.org/QEMU/20200513120147.21443-1-f4bug@amsat.org/
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=960271
the kernel has broken the build of upstream qemu. This made it
into a Debian stable kernel update. This breaks our CI runs almost
completely, when they run with buster.
I spoke to the Debian kernel folks and apparently there is no intent
to fast-track a fix to this. So instead I have made a kernel source
package with the patch from that bug report, and built the
linux-libc-dev package fromk it. The source is here for the time
being:
https://www.chiark.greenend.org.uk/~ijackson/quicksand/2020/libc-kernel-bug.960271/
Deployment note: the source and linux-libc-dev_*.deb are in the
images directory on osstest@test-lab.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
This is a complex interaction between update-grub and the Xen build
system on ARM64. Not sure exactly who to blame but since we have our
own 20_linux_xen bodge, let's wait until we don't.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 19 May 2020 18:13:18 +0000 (19:13 +0100)]
20_linux_xen: Ignore xenpolicy and config files too
"file_is_not_sym" currently only checks for xen-syms. Extend it to
disregard xenpolicy (XSM policy files) and files ending .config (which
are built by the Xen upstream build system in some configurations and
can therefore end up in /boot).
Rename the function accordingly, to "file_is_not_xen_garbage".
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 19 May 2020 18:18:11 +0000 (19:18 +0100)]
20_linux_xen: Adhoc template substitution
This file is a template that various build-time variables get
substituted into. Make thos substitutions by hand (actually, by
copying the values our file for stretch). And rename the file.
So now we are using our file instead of the grub package's. But it is
the same...
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
We are going to want to modify this to support XSM/FLASK and cope with
upstream build outputs.
In this commit we dump the exact file contents across. It's not
effective right now because of the ".in" extension. In fact, the file
is a template.
At the time of writing I am trying to send our substantive changes
upstream via Debian's Gitlab:
https://salsa.debian.org/grub-team/grub/-/merge_requests/18
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
projects / osstest.git / log