On 14:29, Fri, 26 Apr, Gregory Szorc wrote:
On 4/26/2013 2:06 PM, Kartikaya Gupta wrote:On 13-04-26 11:37 , Phil Ringnalda wrote:Unfortunately, engineering is totally indifferent to things like having doubled the cycle time for Win debug browser-chrome since last November.Is there a bug filed for this? I just cranked some of the build.json files through some scripts and got the average time (in seconds) for all the jobs run on the mozilla-central_xp-debug_test-mochitest-browser-chrome builders, and there is in fact a significant increase since November. This makes me think that we need a "resource usage regression" alarm of some sort too. builds-2012-11-01.js: 4063 builds-2012-11-15.js: 4785 builds-2012-12-01.js: 5311 builds-2012-12-15.js: 5563 builds-2013-01-01.js: 6326 builds-2013-01-15.js: 5706 builds-2013-02-01.js: 5823 builds-2013-02-15.js: 6103 builds-2013-03-01.js: 5642 builds-2013-03-15.js: 5187 builds-2013-04-01.js: 5643 builds-2013-04-15.js: 6207Well, wall time will [likely] increase as we write new tests. I'm guessing (OK, really hoping) the number of mochitest files has increased in rough proportion to the wall time? Also, aren't we executing some tests on virtual machines now? On any virtual machine (and especially on EC2), you don't know what else is happening on the physical machine, so CPU and I/O steal are expected to cause variations and slowness in execution time.
Those tests are still on exactly the same hardware. philor points out in https://bugzilla.mozilla.org/show_bug.cgi?id=864085#c0 that the time increase is disproportionate for win7. It would be interesting to look at all the other suites too.
Perhaps a regular report of how much our wall-clock times for builds and different test suite has changed week-over-week would be useful?
That aside, how do we cope with an ever-increasing runtime requirement of tests? Keep adding more chunks?
signature.asc
Description: Digital signature
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
