README.email: Add `Worked example of relevant regression in previous flight'

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

diff --git a/README.email b/README.email
index 5de63ddd161ebf67248094eb7769db43269f5048..e14a8160aac355dda2fa7b8636fb7162b70235b7 100644 (file)
--- a/README.email
+++ b/README.email
@@ -89,6 +89,9 @@ history.  Here are some examples:
       justifiable because they prevent other tests from running and
       can so conceal bugs.)
 
+      See `Worked example of relevant regression in previous flight',
+      below.
+
    fail in 58948 pass in 58965
    fail in 58948 like 37628
 
@@ -159,3 +162,51 @@ X-Osstest-Versions-That:
      tree     revision
 
 `This' is the version being tested, and `That' is the baseline.
+
+
+
+Worked example of relevant regression in previous flight
+--------------------------------------------------------
+
+Suppose two test steps A and B, which normally run in that order:
+  job test-foo
+       A   ./ts-do-some-thing
+       B   ./ts-do-another-thing
+
+Suppose failure of A prevents the execution of B.  (This is the usual
+case where step A precedes step B; normally later steps in a job
+depend on the success of earlier steps, because after an earlier
+failure the testbed state is not necessarily well-defined.)
+
+Now suppose A has an intermittent bug, but B is totally broken.
+
+With our current policy on intermittent bugs[1], we would allow a push
+despite the bug in A.  But we should not allow a push despite B: the
+100% reproducible failure of B should prevent all pushes.
+
+But the bug in B only shows up when A happens to pass.  So the
+heisenbug compensator has to insist on seeing an actual pass of B
+(which in this hypothetical situation, will not occur).
+
+Eg, consider these flights:
+
+  100  is now master  A pass, B pass      pushed
+  200  staging        A pass, B fail      `B REGR. vs 100'
+  201  staging        A fail, B not run   `B fail in 200 REGR. vs 100'
+
+In flight 201, the failure of A is indeed justifiable as a heisenbug
+because it can be seen to succeed in flight 200.  It is the problem
+with B which is actually blocking the push - but that failure is only
+visible in flight 200.
+
+If, contrary to the suppositions above, the failure of B is actually a
+heisenbug, then hopefully eventually both A and then B will happen to
+pass in the same run.  Even if that particular flight has other
+problems, a future evaluation of a test of the same version can use
+that flight's passes of A and B to justify, respectively, whatever
+failures of A and/or B that it comes across.
+
+[1] In principle we could have a different policy: to try to reject
+intermittent bugs.  But it would require a lot of test resources
+because all tests would have to be repeated a lot, and naturally
+intermittent bugs would slip through anyway.