Ian Jackson [Wed, 19 Aug 2020 14:59:09 +0000 (15:59 +0100)]
schema: Provide index on flights by start time
We often use flight number as a proxy for ordering, but this is not
always appropriate and not always done (and sometimes it's a bit of a
bodge).
Provide an index to find flights by start time. This significantly
speeds up the host allocation $equivstatusq query, and the duration
estimator.
(I have tested this by creating a trial index in the production
database. That index can be dropped again, preferably after this
commit makes it to production.)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 19 Aug 2020 12:00:58 +0000 (13:00 +0100)]
host allocation: Memoise duration estimates
We look at our own branch to estimate durations. If somehow we are
one of multiple concurrent flights on this branch with the appropriate
blessing, we don't mind not noticing the doing of our peer flights so
that if our estimates are a bit out of date.
So it is fine to use an estimate no older than our own runtime.
Right now we generate a new duration estimator during each queueing
round, because it contains a statement handle and we must disconnect
from the db while waiting. So the internal memo table gets thrown
away each time and is useless.
To actually memoise, pass our own hash which lives as long as we do.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 19 Aug 2020 11:13:23 +0000 (12:13 +0100)]
resource allocation: Provide OSSTEST_ALLOC_FAKE_PLAN test facility
Set this variable (to a data-plan.final.pl, say) and it becomes
possible to test host allocation programs without actually allocating
anything and without engaging with the queue system.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 19 Aug 2020 12:05:22 +0000 (13:05 +0100)]
ts-hosts-allocate-Executive: Fix broken call to $duration_estimator
The debug subref is passed to the constructor (and indeed we do that).
The final argument to the actual estimator is $uptoincl_testid (but we
didn't say $will_uptoincl_testid, so it is ignored).
The code was wrong, but with no effect. So no functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 10 Aug 2020 15:57:44 +0000 (16:57 +0100)]
history reporting (nfc): Provide cache_set_task_print
This takes a string which gets added to the cache messages. This
will allow us to distinguish the output from different processes
when using parallel by fork.
Nothing sets this yet.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 10 Aug 2020 15:10:50 +0000 (16:10 +0100)]
sg-report-job-history (nfc): Query hosts runvars in one go
Rather than doing one query for each entry in @hostvarcols, do one
query for all the relevant runvars. This is quite a bit faster and
will enable us to use the cache.
This is correct because @hostvarcols was the union of all the host
runvars, so this produces the same answers as the individual queries.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 10 Aug 2020 14:43:33 +0000 (15:43 +0100)]
sg-report-job-history: Refactor "ALL" handling
* Make an explicit entry ALL in @branches, rather than implicitly
processing ALL as well.
* Consequently, put explicit ALL entries in @tasks too, rather than
putting in entries without a branch name.
* Pass ALL to processjobbranch rather than undef, and turn it into
the internally-used undef at the start.
When used with --flight (findflight), this has no functional change.
When used with --job, ALL must now be included in the branch
list passed to --branches. The only in-tree call is with --flight.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 4 Aug 2020 16:23:18 +0000 (17:23 +0100)]
sg-report-job-history: Use one child per report
Rather than one child per job, which then did one report per branch.
This will mean we can use the cache machinery, which is rather global
so wouldn't cope well with processing multiple job history reports
within a process.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Thu, 6 Aug 2020 11:57:31 +0000 (12:57 +0100)]
parallel by fork: Disconnect $dbh_tests as well as undefing it
If the caller is buggy and has statement handles still open, they can
still "work" even if we have thrown away the db handle.
Where, after forking, "work" means "use the same connection in
multiple processes simultaneously, without locking". This could
result in arbitrary crazy nbehaviour (eg, TLS crypto failures).
No functional change with existing callers since they don't have this
bug.
Ian Jackson [Tue, 4 Aug 2020 13:41:09 +0000 (14:41 +0100)]
history reporting: Skip undefined keys
This makes it work if the caller's cached hash contains an key which
is bound to undef.
sg-report-host-history already does this, which currently causes:
Use of uninitialized value $_ in substitution (s///) at Osstest/HistoryReport.pm line 134.
Use of uninitialized value $_ in printf at Osstest/HistoryReport.pm line 135.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Tue, 4 Aug 2020 17:19:57 +0000 (18:19 +0100)]
history reporting (nfc): Bind by name in cacheable_query
cacheable_query used to simply pass $jr->{flight} and ->{job}. But we
want this to be reuseable for other kinds of query, with different
cache keys.
So bind by name: we expect the caller to use :name placeholders in the
query. We can then look through the prepared query parameters, and
fish the corresponding values out of $jr.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Wed, 5 Aug 2020 12:23:41 +0000 (13:23 +0100)]
history reporting (nfc): Refactor to generalise, cache_read_existing
* Introduce @cache_row_key_cols and cache_row_key
* Handle $html_file pathname construction and $read_existing
at the call site.
* Rename cache_read_existing.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Mon, 17 Aug 2020 17:45:43 +0000 (18:45 +0100)]
Tcl: Use tclsh8.6
This is needed to run on buster.
I have checked that tclsh8.6 and TclX works on osstest.test-lab. TclX
seems to be provided by tcl8.4 but work with tcl8.6 (at least on
buster).
Deployment note: hosts running earlier Debian (including
osstest.xs.citrite.net, the Citrix Cambridge instance), may need
OSSTEST_DAEMON_TCLSH=tclsh8.4 or similar in ~/.xen-osstest/settings.
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Ian Jackson [Mon, 10 Aug 2020 15:19:16 +0000 (16:19 +0100)]
SQL: Change "... LIKE '...\_...' ..." to "... LIKE '...\\_...' ..."
Perl's "" quotes, and corresponding <<END constructs, do
\-interpolation, so remove these \ which we carefully added in e7a408dd01184df0a57ae5d9072d15225c52a99f
SQL: Change LIKE E'...\\_...' to LIKE '...\_...'
This is only not a performance problem due to query mismathes with the
available indices, because the test indices are still present in the
Massachusetts instance.
I have verified that this has the intended chanve everywhere by
1. Double-checking that eacbh of these instances is within <<END
or "" or equivalent.
2. Running git-ls-files | xargs perl -i -pe 's/\\\\_/\\_/g'
and manually examining the diff against this patch's parent.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Jim Fehlig [Thu, 23 Jul 2020 21:31:34 +0000 (22:31 +0100)]
Install libtirpc-dev for libvirt builds
The check for XDR support was changed in libvirt commit d7147b3797
to use libtirpc pkg-config instead of complicated AC_CHECK_LIB,
AC_COMPILE_IFELSE, et. al. logic. The libvirt OSSTEST has been
failing since this change hit libvirt.git master. Fix it by adding
libtirpc-dev to the list of 'extra_packages' installed for libvirt
builds.
Signed-off-by: Jim Fehlig <jfehlig@suse.com> Reviewed-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian Jackson [Fri, 31 Jul 2020 10:43:55 +0000 (11:43 +0100)]
duration_estimator: Clarify recentflights query a bit
The condition on r.job is more naturally thought of as a join
condition than a where condition. (This is an inner join, so the
semantics are identical.)
Also, for clarity, swap the flight and job conditions round, so that
the ON clause is a series of r.thing = otherthing.
No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> Reviewed-by: George Dunlap <George.Dunlap@citrix.com>
---
v3: Fix daft syntax error.
v2: New patch.
Ian Jackson [Fri, 24 Jul 2020 18:03:40 +0000 (19:03 +0100)]
cs-bisection-step: Use db_prepare a few times instead of ->do
With $dbh_tests->do(...), we can only get a debug trace of the queries
by using DBI_TRACE which produces voluminous output. Using our own
db_prepare invokes our own debugging.
No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch.
Ian Jackson [Mon, 27 Jul 2020 11:22:51 +0000 (12:22 +0100)]
cs-bisection-step: Generalise qtxt_common_rev_ok
* Make it into a subref which takes a $table argument.
* Change the two references into function calls using the @{...} syntax
* Move the definition earlier in the file
No change to the generated query.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch.
Ian Jackson [Fri, 24 Jul 2020 15:51:28 +0000 (16:51 +0100)]
sg-report-host-history: Rerganisation: Read old logs later
Perhaps at one point something read from these logs influenced the db
query for thye flights range, but that is no longer the case and it
doesn't seem likely to need to come back.
We want to move the per-host stuff together.
No functional change.
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
Now that we have an index, there is a faster way for the db to do this
query: via that index. But it doesn't like to do that if be aggregate
the queries. Experimentally, doing this query separately once per
host is significantly faster.
Also, later, it will allow us to parallelise this work.
So, we undo that. (Not by reverting, though.)
Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Use proper \ escaping for underscores in LIKE
projects / osstest.git / log