A downside to chunking such that skipped tests are taken into account is
that it will make some chunks take longer to run than others. This
negatively impacts the scheduling algorithms and ultimately increases
infrastructure load.
The effects are especially noticeable on new platforms/applications that
disable large swathes of tests to begin with (e.g b2g, android). For
example, while trying to get reftests running on B2G emulators I saw
cases where one chunk took 20 minutes while the next took ~120. I
actually wrote a patch to do the opposite of what you are suggesting
here (to not count skipped tests while chunking).
Correct me if I'm wrong, but I think the issue you are trying to solve
is how to figure out what try syntax is needed to run a certain test. I
think there are better ways to solve this problem. I.e a more expressive
try syntax such that you can just specify the test or directory of tests
to run and not care about what chunk it is in. Bonus points if we can
configure mozharness to run only those tests and ignore all other tests
that happen to be in the same chunk.
-Andrew
On 01/21/2014 12:05 PM, Ehsan Akhgari wrote:
It seems to me like the splitting algorithm for mochitests gives us
different chunking results on different platforms, see this test failure
for example: <
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=746018b05d67>. Is this
expected? If not, is this easy to fix so that a given test always falls
inside the same chunk if the number of chunks are equal on two different
platforms?
Thanks!
--
Ehsan
<http://ehsanakhgari.org/>
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
