On Fri, 2015-11-27 at 15:38 +0000, Ian Jackson wrote: > Signed-off-by: Ian Jackson <ian.jack...@eu.citrix.com> > CC: Jan Beulich <jbeul...@suse.com>
Acked-by: Ian Campbell <ian.campb...@citrix.com> > --- > README.email | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 51 insertions(+) > > diff --git a/README.email b/README.email > index 5de63dd..e14a816 100644 > --- a/README.email > +++ b/README.email > @@ -89,6 +89,9 @@ history. Here are some examples: > justifiable because they prevent other tests from running and > can so conceal bugs.) > > + See `Worked example of relevant regression in previous flight', > + below. > + > fail in 58948 pass in 58965 > fail in 58948 like 37628 > > @@ -159,3 +162,51 @@ X-Osstest-Versions-That: > tree revision > > `This' is the version being tested, and `That' is the baseline. > + > + > + > +Worked example of relevant regression in previous flight > +-------------------------------------------------------- > + > +Suppose two test steps A and B, which normally run in that order: > + job test-foo > + A ./ts-do-some-thing > + B ./ts-do-another-thing > + > +Suppose failure of A prevents the execution of B. (This is the usual > +case where step A precedes step B; normally later steps in a job > +depend on the success of earlier steps, because after an earlier > +failure the testbed state is not necessarily well-defined.) > + > +Now suppose A has an intermittent bug, but B is totally broken. > + > +With our current policy on intermittent bugs[1], we would allow a push > +despite the bug in A. But we should not allow a push despite B: the > +100% reproducible failure of B should prevent all pushes. > + > +But the bug in B only shows up when A happens to pass. So the > +heisenbug compensator has to insist on seeing an actual pass of B > +(which in this hypothetical situation, will not occur). > + > +Eg, consider these flights: > + > + 100 is now master A pass, B pass pushed > + 200 staging A pass, B fail `B REGR. vs 100' > + 201 staging A fail, B not run `B fail in 200 REGR. vs 100' > + > +In flight 201, the failure of A is indeed justifiable as a heisenbug > +because it can be seen to succeed in flight 200. It is the problem > +with B which is actually blocking the push - but that failure is only > +visible in flight 200. > + > +If, contrary to the suppositions above, the failure of B is actually a > +heisenbug, then hopefully eventually both A and then B will happen to > +pass in the same run. Even if that particular flight has other > +problems, a future evaluation of a test of the same version can use > +that flight's passes of A and B to justify, respectively, whatever > +failures of A and/or B that it comes across. > + > +[1] In principle we could have a different policy: to try to reject > +intermittent bugs. But it would require a lot of test resources > +because all tests would have to be repeated a lot, and naturally > +intermittent bugs would slip through anyway. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
