A proposal to get this fixed would be immense and ties into the projects
we are doing to help keep engineers productive. Things like parallel
testing[1] should be done to as many test suites as possible, a smoke
test to get feedback loops tighter.
The main one for me is getting easily accessible data of each push so we
can make informed decisions and target issues properly instead of guessing.
My main point is landing patches on Inbound that burn the tree have the
affect of using resources that could have been used for try. At the
moment I don't have the data but times for try always seem quite some
time. The other issue is loss of productivity of an engineer who then
pulls from a broken revision not knowing why its broken and looks to see
if it is their patch.
A lot of these costs at the moment are not easily accessible, though I
am trying to get them. The A*Team has been trying to, on top of its
normal tasks, to try figure out which tests fail on which platform and
try see if we can come up with a smoke test suite[2] (see Ed Morley's
email about it in this thread too).
[1] http://www.mihneadb.net/post/parallelizing-a-test-harness/
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=863838
On 05/11/2013 14:57, Kyle Huey wrote:
On Tue, Nov 5, 2013 at 10:44 PM, David Burns <dbu...@mozilla.com
<mailto:dbu...@mozilla.com>> wrote:
We appear to be doing 1 backout for every 15 pushes on a rough
average[4]. This number I am sure you can all agree is far too
high especially if we think about the figures that John O'Duinn
suggests[5] for the cost of each push for running and testing.
With the offending patch + backout we are using 508 computing
hours for essentially doing no changes to the tree and then we do
another 254 computing hours for the fixed reland. Note the that
the 508 hours doesn't include retriggers done by the Sheriffs to
see if it is intermittent or not.
This is a lot of wasted effort when we should be striving to get
patches to stick first time. Let's see if we can try make this
figure 1 in 30 patches getting backed out.
What is your proposal for doing that? What are the costs involved?
It isn't very useful to say X is bad, let's not do X, without looking
at what it costs to not do X.
To give one hypothetical example, if it requires just two additional
full try pushes to avoid one backout, we haven't actually saved any
computing time.
- Kyle
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
