On 4/26/2013 2:06 PM, Kartikaya Gupta wrote: > On 13-04-26 11:37 , Phil Ringnalda wrote: >> Unfortunately, engineering is totally indifferent to >> things like having doubled the cycle time for Win debug browser-chrome >> since last November. >> > > Is there a bug filed for this? I just cranked some of the build.json > files through some scripts and got the average time (in seconds) for > all the jobs run on the > mozilla-central_xp-debug_test-mochitest-browser-chrome builders, and > there is in fact a significant increase since November. This makes me > think that we need a "resource usage regression" alarm of some sort too. > > builds-2012-11-01.js: 4063 > builds-2012-11-15.js: 4785 > builds-2012-12-01.js: 5311 > builds-2012-12-15.js: 5563 > builds-2013-01-01.js: 6326 > builds-2013-01-15.js: 5706 > builds-2013-02-01.js: 5823 > builds-2013-02-15.js: 6103 > builds-2013-03-01.js: 5642 > builds-2013-03-15.js: 5187 > builds-2013-04-01.js: 5643 > builds-2013-04-15.js: 6207
Well, wall time will [likely] increase as we write new tests. I'm guessing (OK, really hoping) the number of mochitest files has increased in rough proportion to the wall time? Also, aren't we executing some tests on virtual machines now? On any virtual machine (and especially on EC2), you don't know what else is happening on the physical machine, so CPU and I/O steal are expected to cause variations and slowness in execution time. Speaking of resource usage, I've filed bug 859573 to have system resource "counters" reported as part of jobs. That way, we can have a high-level handle on whether our CPU efficiency is increasing/decreasing over time. I'd argue that we should strive for 100% CPU saturation on every slave (for most jobs) otherwise those CPU cycles are lost forever and we've wasted capacity. But, that's arguably a conversation for another thread. While I don't have numbers off hand, one of the things I noticed was the wall time of the various test chunks isn't as balanced as it should be. In particular, bc tests seem to be a long pole. Perhaps we should split them into bc-1 and bc-2? Along that vein, perhaps we could combine some of the regular mochitest jobs, as they don't seem to take too long to execute. Who makes these kinds of decisions? On the subject of mochitests, I think we should really pound home the message that mochitests should be avoided if possible. If you can move more "business logic" into JSMs and test with xpcshell tests and only write mochitests for the code that exists in the browser, that's a net win (xpcshell tests are lighter weight and easier to run in parallel). This would likely involve a huge shift in the way FX Team (and others) write code and tests, so I don't expect it will be an easy sell. But, it's a discussion we should have because the impact on test execution times could be drastic. _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
