Re: Some data on mozilla-inbound

On 14:29, Fri, 26 Apr, Gregory Szorc wrote: On 4/26/2013 2:06 PM, Kartikaya Gupta wrote: On 13-04-26 11:37 , Phil Ringnalda wrote: Unfortunately, engineering is totally indifferent to things like having doubled the cycle time for Win debug browser-chrome since last November. Is there a bug

Re: Some data on mozilla-inbound

On 4/26/2013 2:06 PM, Kartikaya Gupta wrote: > On 13-04-26 11:37 , Phil Ringnalda wrote: >> Unfortunately, engineering is totally indifferent to >> things like having doubled the cycle time for Win debug browser-chrome >> since last November. >> > > Is there a bug filed for this? I just cranked so

Re: Some data on mozilla-inbound

Bug 864085 On Fri, Apr 26, 2013 at 2:06 PM, Kartikaya Gupta wrote: > On 13-04-26 11:37 , Phil Ringnalda wrote: >> >> Unfortunately, engineering is totally indifferent to >> things like having doubled the cycle time for Win debug browser-chrome >> since last November. >> > > Is there a bug filed

Re: Some data on mozilla-inbound

On 13-04-26 11:37 , Phil Ringnalda wrote: Unfortunately, engineering is totally indifferent to things like having doubled the cycle time for Win debug browser-chrome since last November. Is there a bug filed for this? I just cranked some of the build.json files through some scripts and got t

Re: Some data on mozilla-inbound

On 4/26/13 8:25 AM, Wesley Johnston wrote: > Maybe. I started to avoid it if possible around then, but almost 4 hours for > results still is basically unusable. Tell me about it - that's actually the same as the end-to-end on inbound/central. Unfortunately, engineering is totally indifferent to t

Re: Some data on mozilla-inbound

Maybe. I started to avoid it if possible around then, but almost 4 hours for results still is basically unusable. - Wes - Original Message - From: "Phil Ringnalda" To: dev-platform@lists.mozilla.org Sent: Friday, April 26, 2013 8:01:25 AM Subject: Re: Some data on mozilla-inb

Re: Some data on mozilla-inbound

On 4/25/13 4:47 PM, Wesley Johnston wrote: > Requesting one set of tests on one platform is a 6-10 hour turnaround for me. That's surprising. https://tbpl.mozilla.org/?tree=Try&rev=9d1daf69061d was a midday -b do -p all -u all with a 3 hour 40 minute end-to-end. Or did you mean, as a great many p

Re: Some data on mozilla-inbound

Requesting one set of tests on one platform is a 6-10 hour turnaround for me. - Original Message - From: "Neil" To: dev-platform@lists.mozilla.org Sent: Thursday, April 25, 2013 4:36:02 PM Subject: Re: Some data on mozilla-inbound Justin Lebar wrote: >Note that we don&

Re: Some data on mozilla-inbound

Justin Lebar wrote: Note that we don't have enough capacity to turn around current try requests within a reasonable amount of time. Is this because people are requesting too much because try chooser simply isn't sufficiently descriptive for what people want? -- Warning: May contain traces o

Re: Some data on mozilla-inbound

It seems as though fixing this bug for releng: https://bugzilla.mozilla.org/show_bug.cgi?id=793989 and adding a UI for self-serve (etc..) would allow us to build/test on one platform, then request new builds/tests for other platforms if our problem is solved.

Re: Some data on mozilla-inbound

With extremely limited experience of using try, I know that I would have at times set a flag "stop as soon as you hit a first red on a platform". So, I really like Chris' idea below, as a manual workaround, and a more powerful solution for that. Easier said than done, I imagine... Milan On

Re: Some data on mozilla-inbound

On 25 April 2013 16:11:59, Justin Lebar wrote: Which specific proposals should we start with? As you say, there are dozens of ideas out there, none with any kind of consensus behind them. If a preponderance of options is actually the only thing standing in the way of serious and timely work bei

Re: Some data on mozilla-inbound

> Which specific proposals should we start with? As you say, there are dozens > of ideas out there, none with any kind of consensus behind them. If a preponderance of options is actually the only thing standing in the way of serious and timely work being done by releng here, I would be more than h

Re: Some data on mozilla-inbound

On 01:48, Thu, 25 Apr, Justin Lebar wrote: One idea might be to give developers feedback on the consequences of a particular push, e.g. the AWS cost, a proxy for "time during which developers couldn't push" or some other measurable metric. Right now each push probably "feels" as "expensive" as e

Re: Some data on mozilla-inbound

On 2013-04-25 2:42 AM, Phil Ringnalda wrote: On 4/24/13 9:50 PM, Ehsan Akhgari wrote: No. But that's not what I was talking about. Whether something lands directly on try is a judgement call, and some people may be better at it than others. As someone who has stopped using try server as a rul

Re: Some data on mozilla-inbound

On 4/24/13 9:50 PM, Ehsan Akhgari wrote: > No. But that's not what I was talking about. Whether something lands > directly on try is a judgement call, and some people may be better at it > than others. As someone who has stopped using try server as a rule > (because of the excessive wait times t

Re: Some data on mozilla-inbound

On 4/22/13 12:54 PM, Kartikaya Gupta wrote: > I looked at all the build.json files [4] from the 6th of April to the > 17th of April and pulled out all the jobs that corresponding to the > "push" changesets in my range above. For this set of 553 changesets, > there were 500 (exactly!) distinct "buil

Re: Some data on mozilla-inbound

> One idea might be to give developers feedback on the consequences of a > particular push, e.g. the AWS cost, a proxy for "time during which > developers couldn't push" or some other measurable metric. Right now > each push probably "feels" as "expensive" as every other. For tryserver, I propose

Re: Some data on mozilla-inbound

On 2013-04-25 1:02 AM, David Ascher wrote: The messaging around this should not be to tell people "always test on try". It should be to help them figure out how to make better judgement calls on this. This is a skill that people develop and are not born with, and without data i

Re: Some data on mozilla-inbound

> The messaging around this should not be to tell people "always test on > try". It should be to help them figure out how to make better judgement > calls on this. This is a skill that people develop and are not born with, > and without data it's hard an an individual to judge how good I'm at tha

Re: Some data on mozilla-inbound

On 2013-04-24 9:14 AM, Ben Hearsum wrote: On 04/23/13 10:21 PM, Kartikaya Gupta wrote: On 13-04-23 19:21 , Nicholas Nethercote wrote: - The 'inbound was closed for 15.3068% of the total time due to "bustage"' number is an underestimate, in one sense. When inbound is closed at 10am California t

Re: Some data on mozilla-inbound

On 2013-04-23 1:17 PM, Ed Morley wrote: On 23/04/2013 17:28, Kartikaya Gupta wrote: On 13-04-23 00:39 , Ehsan Akhgari wrote: How hard would it be to gather a list of the total number of patches being backed out plus the amount of time that we spent building/testing those, hopefully in a style s

Re: Some data on mozilla-inbound

On 2013-04-23 12:05 PM, Justin Lebar wrote: The ratio of things landed on inbound which turn out to busted is really worrying On the one hand, we're told not to push to try too much, because that wastes resources. On the other hand, we're told not to burn m-i, because that wastes resources.

Re: Some data on mozilla-inbound

On 04/23/13 10:21 PM, Kartikaya Gupta wrote: > On 13-04-23 19:21 , Nicholas Nethercote wrote: >> - The 'inbound was closed for 15.3068% of the total time due to >> "bustage"' number is an underestimate, in one sense. When inbound is >> closed at 10am California time, it's a lot more inconvenient t

Re: Some data on mozilla-inbound

On 13-04-23 19:21 , Nicholas Nethercote wrote: - The 'inbound was closed for 15.3068% of the total time due to "bustage"' number is an underestimate, in one sense. When inbound is closed at 10am California time, it's a lot more inconvenient to developers than when it's busted at midnight Califor

Re: Some data on mozilla-inbound

On Wed, Apr 24, 2013 at 11:21 AM, Nicholas Nethercote < n.netherc...@gmail.com> wrote: > - The 'inbound was closed for 15.3068% of the total time due to > "bustage"' number is an underestimate, in one sense. When inbound is > closed at 10am California time, it's a lot more inconvenient to > devel

Re: Some data on mozilla-inbound

On Mon, Apr 22, 2013 at 12:54 PM, Kartikaya Gupta wrote: > TL;DR: > * Inbound is closed 25% of the time > * Turning off coalescing could increase resource usage by up to 60% (but > probably less than this). > * We spend 24% of our machine resources on changes that are later backed > out, or change

Re: Some data on mozilla-inbound

On 2013-04-23 12:50 AM, Justin Lebar wrote: The ratio of things landed on inbound which turn out to busted is really worrying * 116 of the 916 changesets (12.66%) were backed out If 13% is "really worrying", what do you think our goal should be? Less than that? It's really hard to come up

Re: Some data on mozilla-inbound

On 4/23/13 6:35 PM, Kartikaya Gupta wrote: On 13-04-23 03:57 , Axel Hecht wrote: Do we know how many of these have been pushed to try, and passed/compiled what they'd fail later? I haven't looked at this. It would be useful to know but short of pulling patches and using some similarity heurist

Re: Some data on mozilla-inbound

On 4/23/13 1:17 PM, David Keeler wrote: I would like to know a bit more about this. Is our list of supported toolchains so diverse that building with one version versus another will report so many false positives as to be useless? Yes. For example a typical clang+ccache build of the tree with

Re: Some data on mozilla-inbound

On 23/04/2013 17:28, Kartikaya Gupta wrote: On 13-04-23 00:39 , Ehsan Akhgari wrote: How hard would it be to gather a list of the total number of patches being backed out plus the amount of time that we spent building/testing those, hopefully in a style similar to

Re: Some data on mozilla-inbound

On 04/23/13 02:17, Ed Morley wrote: > On 23 April 2013 09:58:41, Neil wrote: >> Hopefully a push never burns all platforms because the developer tried >> it locally first, but stranger things have happened! > > This actually happens quite often. On occasion it's due to warnings as > errors (switch

Re: Some data on mozilla-inbound

On Tue, Apr 23, 2013 at 9:28 AM, Kartikaya Gupta wrote: > Not trivial, but not too difficult either. Do we have any evidence to show > that the try highscores page has made an impact in reducing unnecessary try > usage? It's been used by people like Ed Morley to reach out to individual developers

Re: Some data on mozilla-inbound

On Tue, Apr 23, 2013 at 8:41 AM, Chris AtLee wrote: > We've considered enforcing this using some cryptographic token. After you > push to try and get good results, the system gives you a token you need to > include in your commit to m-i. Sounds like the goal of this kind of solution would be to e

Re: Some data on mozilla-inbound

On 13-04-23 03:57 , Axel Hecht wrote: Do we know how many of these have been pushed to try, and passed/compiled what they'd fail later? I haven't looked at this. It would be useful to know but short of pulling patches and using some similarity heuristic or manually examining patches I can't t

Re: Some data on mozilla-inbound

On 13-04-23 00:39 , Ehsan Akhgari wrote: How hard would it be to gather a list of the total number of patches being backed out plus the amount of time that we spent building/testing those, hopefully in a style similar to ? Not trivia

Re: Some data on mozilla-inbound

On 13-04-23 11:41 , Chris AtLee wrote: We've considered enforcing this using some cryptographic token. After you push to try and get good results, the system gives you a token you need to include in your commit to m-i. ... or you could just merge the cset directly from try to m-i or m-c. (i.e.

Re: Some data on mozilla-inbound

>> The ratio of things landed on inbound which turn out to busted is really >> worrying On the one hand, we're told not to push to try too much, because that wastes resources. On the other hand, we're told not to burn m-i, because that wastes resources. Should we be surprised when people don't g

Re: Some data on mozilla-inbound

On 16:34, Tue, 23 Apr, Gervase Markham wrote: On 23/04/13 10:17, Ed Morley wrote: Given that local machine time scales linearly with the rate at which we hire devs (unlike our automation capacity), I think we need to work out why (some) people aren't doing things like compiling locally and runni

Re: Some data on mozilla-inbound

On 23/04/13 10:17, Ed Morley wrote: > Given that local machine time scales linearly with the rate at which we > hire devs (unlike our automation capacity), I think we need to work out > why (some) people aren't doing things like compiling locally and running > their team's directory of tests before

Re: Some data on mozilla-inbound

On 23 April 2013 09:58:41, Neil wrote: Hopefully a push never burns all platforms because the developer tried it locally first, but stranger things have happened! This actually happens quite often. On occasion it's due to warnings as errors (switched off by default on local machines due to too

Re: Some data on mozilla-inbound

On 23/4/13 09:58, Neil wrote: Kartikaya Gupta wrote: The vast majority of changesets that are backed out from inbound are detectable on a try push Hopefully a push never burns all platforms because the developer tried it locally first, but stranger things have happened! But what I'm most inte

Re: Some data on mozilla-inbound

Kartikaya Gupta wrote: The vast majority of changesets that are backed out from inbound are detectable on a try push Hopefully a push never burns all platforms because the developer tried it locally first, but stranger things have happened! But what I'm most interested in is whether patches

Re: Some data on mozilla-inbound

On 4/22/13 9:54 PM, Kartikaya Gupta wrote: TL;DR: * Inbound is closed 25% of the time * Turning off coalescing could increase resource usage by up to 60% (but probably less than this). * We spend 24% of our machine resources on changes that are later backed out, or changes that are doing the back

Re: Some data on mozilla-inbound

> The ratio of things landed on inbound which turn out to busted is really > worrying > * 116 of the 916 changesets (12.66%) were backed out If 13% is "really worrying", what do you think our goal should be? On Tue, Apr 23, 2013 at 12:39 AM, Ehsan Akhgari wrote: > This was a fantastic read, it

Re: Some data on mozilla-inbound

This was a fantastic read, it almost made me shed happy tears! Thanks a lot kats for doing this. The ratio of things landed on inbound which turn out to busted is really worrying, and it might be an indicator that (some?) developers have a poor judgement on how safe their patches are. How ha

Some data on mozilla-inbound

TL;DR: * Inbound is closed 25% of the time * Turning off coalescing could increase resource usage by up to 60% (but probably less than this). * We spend 24% of our machine resources on changes that are later backed out, or changes that are doing the backout * The vast majority of changesets that