It will be handled just like coalesced jobs today: sheriffs will
backfill the missing data, and backout only the offender.
An illustration might help. Today we might have something like this,
for a given job:
linux64-debug win7-debug osx8-debug
commit 1 pass pass pass
commit 2 pass pass pass
commit 3 pass fail pass
commit 4 pass fail pass
In this case (assuming the two failures are the same), it's easy for
sheriffs to see that commit 3 is the culprit and the one that needs to
be backed out.
During the experiment, we might see something like this:
linux64-debug win7-debug osx8-debug
commit 1 pass pass pass
commit 2 pass not run not run
commit 3 pass fail pass
commit 4 pass not run not run
Here, it isn't obvious whether the problem is caused by commit 2 or
commit 3. (This situation already occurs today because of "random"
coalescing.)
In this case, the sheriffs will backfill missing test data, so we might see:
linux64-debug win7-debug osx8-debug
commit 1 pass pass pass
commit 2 pass pass not run
commit 3 pass fail pass
commit 4 pass fail not run
...and then they have enough data to determine that commit 3 (and not
commit 2) is to blame, and can take the appropriate action.
In summary, the sheriffs won't be backing out extra commits because of
the coalescing, and it remains the sheriffs' job to backfill tests when
they determine they need to do so in order to bisect a failure. We
aren't placing any extra burden on developers with this experiment, and
part of the reason for this experiment is to determine how much of an
extra burden this is for the sheriffs.
Jonathan
On 8/21/2014 3:03 PM, Jonas Sicking wrote:
What will be the policy if a test fails and it's unclear which push
caused the regression? Is it the sheriff's job, or the people who
pushed's job to figure out which push was the culprit and make sure
that that push gets backed out?
I.e. if 4 pushes land between two testruns, and we see a regression,
will the 4 pushes be backed out? Or will sheriffs run the missing
tests and only back out the offending push?
/ Jonas
On Thu, Aug 21, 2014 at 10:50 AM, Jonathan Griffin <jgrif...@mozilla.com> wrote:
Thanks Ed. To paraphrase, no test coverage is being lost here, we're just
being a little more deliberate with job coalescing. All tests will be run
on all platforms (including debug tests) on a commit before a merge to m-c.
Jonathan
On 8/21/2014 9:35 AM, Ed Morley wrote:
I think much of the pushback in this thread is due to a misunderstanding
of some combination of:
* our current buildbot scheduling
* the proposal
* how trees are sheriffed and merged
To clarify:
1) We already have coalescing [*] of jobs on all trees apart from try.
2) This coalescing means that all jobs are still run at some point, but
just may not run on every push.
3) When failures are detected, coalescing means that regression ranges are
larger and so sometimes result in longer tree integration repo closures,
whilst the sheriffs force trigger jobs on the revisions that did not
originally run them.
4) When merging into mozilla-central, sheriffs ensure that all jobs are
green - including those that got coalesced and those that are only scheduled
periodically (eg non-unified & PGO builds are only run every 3 hours). (This
is a fairly manual process currently, but better tooling should be possible
with treeherder).
5) This proposal does not mean debug-only issues are somehow not worth
acting on or that they'll end up shipped/on mozilla-central, thanks to #4.
6) This proposal is purely trying to make existing coalescing (#1/#2) more
intelligent, to ensure that we expend the finite amount of machine time we
have at present on the most appropriate jobs at each point, in order to
reduce the impact of #3.
Fwiw I'm on the fence as to whether the algorithm suggested in this
proposal is the most effective way to aid with #3 - however it's worth
trying to find out.
Best wishes,
Ed
[*] Collapsing of pending jobs of the same type, when the queue size is
greater than 1.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
