On 8/31/2012 8:46 AM, Justin Lebar wrote:
There are extremely non-stable Talos tests, and relatively stable ones.
Let's focus on the relatively stable ones.
It's not exclusively a question of noise in the tests. Even
regressions in stable tests are sometimes hard to track down. I spent
two months trying to figure out why I could not reproduce a Dromaeo
regression I saw on m-i using try, and eventually gave up (bug
653961).
It's great if we try to track down this mysterious 5% startup
regression. We shouldn't ignore important regressions. But what I
object to is the idea that if I regress Dromaeo DOM by 2%, I'm
automatically backed out and prevented from doing any work until I
prove that the problem is I changed filename somewhere.
I think we can solve this by having go-to people for Talos performance
help. If you have spent a sufficient amount of time tracking down a
regression, you should be able to ask someone like Rafael* to help.
Same people should also have the final say on whether we should give up
on tracking down a regression.
Letting every individual committer decide 'this is too hard to track
down' makes it too easy to let the regression slide and is one of the
reasons we ended up in the current situation.
Taras
* I'm using Rafael as a hypothetical go-to person because he is
participating in this thread. We have other people in the project who
are well-versed in tracking down odd perf regressions.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
