On Fri, 2015-11-27 at 12:02 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112:
> regressions - FAIL"):
> > On Fri, 2015-11-27 at 01:18 -0700, Jan Beulich wrote:
> > > Neither of these failed in this flight, and there's nothing else
> > > blocking
> > > the push. Why did this not result in a push then? Or in other words
> > > why do the failures in earlier flights get considered a reason not to
> > > push?
> >
> > @Ian, README.email covers lots of these kinds of patterns, but not this
> > specific one.
>
> See below for proposed docs patch to explain the general meaning of
> `fail in X REGR. vs. Y'.
>
>
> > > > build-i386 5 xen-build fail in 65062 REGR. vs. 63449
>
> This is completely explained below, I think.
>
> > > > test-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 16 guest-
> > > > localmigrate/x10 fail in 65088 REGR. vs. 63449
>
> As explained below, in 65112 this step did not run because the earlier
> step `guest-localmigrate' failed:
> http://logs.test-lab.xenproject.org/osstest/logs/65112/test-amd64-
> amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/info.html
Would it be possible to arrange for "blocked" to appear somewhere in the
results for the job? e.g. "blocked fail in XXX REGR. vs. YYY". README.email
says "The results normally start with the result in this flight" and I
think this would be in keeping with that.
Otherwise I think people naturally tend to just read the "and are blocking"
section and forget to consider that non-blocking stuff further down may
have (tolerably) failed but then blocking something else which is then
blocking the push.
> The fact that we have both `guest-localmigrate' and
> `guest-localmigrate/x10' isn't ideal because it hides from the
> heisenbug compensator that these are actually the same underlying
> test. Maybe it is time now to rename `guest-localmigrate/x10' to
> `guest-localmigrate' and abolish the latter.
I think this would be a good idea.
> From 987dd088192f9f94c59beeddc073cecaad76a24e Mon Sep 17 00:00:00 2001
> From: Ian Jackson <ian.jack...@eu.citrix.com>
> Date: Fri, 27 Nov 2015 11:36:05 +0000
> Subject: [OSSTEST PATCH] README.email: Document `fail in 58948 REGR. vs.
> 63449'
>
> Signed-off-by: Ian Jackson <ian.jack...@eu.citrix.com>
Acked-by: Ian Campbell <ian.campb...@citrix.com>
> ---
> README.email | 18 ++++++++++++++++++
> 1 file changed, 18 insertions(+)
>
> diff --git a/README.email b/README.email
> index 992a574..40df71a 100644
> --- a/README.email
> +++ b/README.email
> @@ -71,6 +71,24 @@ history. Here are some examples:
> detect regressions of this test. Perhaps the test has been
> recently introduced.
>
> + fail in 58948 REGR. vs. 63449
> +
> + The results processor used 58948 (another flight testing the
> + just-tested version) to convince itself that some other test
> + failure is intermittent. Look for other references to 58948 in
> + the report to see which those other test failures are.
> +
> + However, in 58948, there were further failures. In particular,
> + the step being reported here failed, and that failure could not
> + in turn be justified.
> +
> + If this further failure is in a test job, this is usually
> + because the reported step did not run at all in the most recent
> + flight, usually because it was blocked by an earlier failure.
> + (Intermittent build job failures are never considered
> + justifiable because they prevent other tests from running and
> + can so conceal bugs.)
> +
> fail in 58948 pass in 58965
> fail in 58948 like 37628
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
