Database locking: Perl: Retry all deadlocks in PostgreSQL

Database locking: Perl: Retry all deadlocks in PostgreSQL

Previously we would retry all COMMITs but nothing else.  This is
correct for SQLite3 but not for PostgreSQL.

We got away with it before because of the heavyweight locking of even
long-running read-only transactions, but now the LOCK TABLEs can fail
(at least in a mixed-version system, and perhaps even in a system with
only new code).

So: cover all of the database work in db_retry with the eval, and
explicitly ask the JobDB adaptation layer (via a new need_retry
method) whether to go around again.  We tell the JobDB layer whether
the problem was during commit, so that we can avoid making any overall
semantic change to the interaction with SQLite3.

In the PostgreSQL case, the db handle can be asked whether there was
an error and what the error code was.  Deadlock has its own error
code.

(One side effect here is that db_retry_retry, which sets
$db_retry_stop='retry', is now no longer affected by the retry count
in db_retry.  But there are no callers and that may be more right
anyway.  db_retry_abort always exits the loop, as before.)

I have tested this with the following rune:

 OSSTEST_CONFIG=/u/iwj/.xen-osstest/config:local-config.test-database_iwj perl -w -MData::Dumper -e 'use strict; use Osstest::Executive; use Osstest; csreadconfig(); print Dumper($dbh_tests->{AutoCommit}); eval { $dbh_tests->do("BOGUS"); }; db_begin_work($dbh_tests, [qw(flights resources)])'

adding a sleep(2) to the loop Osstest::JobDB::Executive::begin_work,
and running a second copy of the rune with the tables to lock in the
other order.

Acked-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Mention db_retry_retry in commit message.