*tl;dr**: Your |mach try auto| pushes can now precisely schedule relevant
manifests, catching more regressions with fewer resources.*
I'm happy to announce an important new capability that has been added to
our CI infrastructure: scheduling specific test manifests.
We've been building a machine learning algorithm to determine which tests
are most relevant to a given push. This is now running on autoland, as well
as when you push to try with |mach try auto|. Previously, the biggest
limitation of this system was that test manifests were already assigned to
tasks by the time we could run the ML algorithm. So we'd have a list of
important manifests, then run the chunks that contain those manifests. The
drawback was we'd also run all the other unimportant manifests that
happened to be in the same chunk as the important one.
As of late last week, |mach try auto| is now running in "manifest
scheduling" mode. This means the ML algorithm is running earlier in the
process and is actually creating the test chunks based on how many
important manifests there are. Therefore we are only running manifests that
are relevant to your push and are avoiding the collateral damage caused by
all the "unimportant" ones. As part of this change I've tweaked the minimum
importance threshold downward, meaning the algorithm classifies a greater
number of manifests as "important". So |mach try auto| pushes will schedule
a greater number of relevant manifests, while simultaneously scheduling
fewer manifests over all.
One thing to note is that the chunks in your |mach try auto| pushes will
not bear any resemblance to the chunks that are created on autoland. In
fact, they may bear no resemblance to the previous |mach try auto| push you
made, even with similar changes. If you need to figure out where a given
test ran you can use Treeherder's test path filter:
1. Click the filter icon near the top right in Treeherder (next to
coloured circles)
2. Select "test path" from the drop down
3. Type out the test path you are looking for
4. Click "Add"
Please note that reftest isn't supported in this filter yet, and WPT
requires test identifiers (e.g `/foo` instead of
`testing/web-platform/tests/foo`).
*What's Next?*
There is a lot of work still coming down the pipeline. First and foremost,
we want to enable "manifest scheduling" on autoland in addition to |mach
try auto|. There are certain problems that exist on autoland (like sheriff
backfilling) that we didn't need to worry about on try. Next we'll support
manifest scheduling on all test suites (the major ones that are missing
include reftest and derivatives, WPT reftests and mochitest-webgl). Last
but not least, we're going to continue working through all of the feedback
we've been gathering over the past weeks. Thanks to everyone who has left
feedback around |mach try auto|, it's been invaluable!
In case this post prompts you to try it out (or to revisit it again), then
please file bugs and suggestions under `Firefox Build System :: Try`, or
ping us in #test-selection on Matrix.
Cheers!
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
