On 16-02-10 04:58 AM, Marco Bonardo wrote: > On Wed, Feb 10, 2016 at 10:30 AM, James Graham <ja...@hoppipolla.co.uk> > wrote: > >> FWIW I think it's closer to the truth to say that these tests are not set >> up to be performance regression tests >> > > Right, but this was just one of the aspects that was pointed out. I think > the performance analysis is not much about catching regressions (even if > that can effectively happen), rather figuring why a test takes 90 seconds > instead of 10, is that a problem that may affect end users too? > The larger we set the threshold, the less people are likely to investigate > the reasons the test takes so much, cause there will be no bugs filed about > the problem. It will just go unnoticed. > Other reasons are bound to costs imo. Developers time is a cost, if > everyone starts writing tests taking more time, cause now the timeout is > much larger, we may double the time to run tests locally or to get results > out of Try. > Finally, bumping timeouts isn't a final solution, that means in a few years > we may be back rediscussing another bump. > > I'm not saying bumping the timeout is wrong, I'm saying we should also > evaluate a long term strategy to avoid the downsides. Maybe we should have > something like the orange factor tracking tests runtime (in all the > harnesses) and automatically filing bugs in the appropriate component when > some test goes over a threshold? >
In this case it would be better to have a tool that keeps track of test runtimes and report regressions. ActiveData has the data to do this. -- Zambrano Gasparnian, Armen Automation & Tools Engineer http://armenzg.blogspot.ca _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
