This is very exciting on a number of fronts: reduced CI spend, detecting more defects, and reduced cognitive overhead for developers. Thanks to Andrew and everyone else involved for getting us here!
On Mon, Jun 8, 2020 at 9:14 AM Andrew Halberstadt <a...@mozilla.com> wrote: > *tl;dr**: Your |mach try auto| pushes can now precisely schedule relevant > manifests, catching more regressions with fewer resources.* > > I'm happy to announce an important new capability that has been added to > our CI infrastructure: scheduling specific test manifests. > > We've been building a machine learning algorithm to determine which tests > are most relevant to a given push. This is now running on autoland, as well > as when you push to try with |mach try auto|. Previously, the biggest > limitation of this system was that test manifests were already assigned to > tasks by the time we could run the ML algorithm. So we'd have a list of > important manifests, then run the chunks that contain those manifests. The > drawback was we'd also run all the other unimportant manifests that > happened to be in the same chunk as the important one. > > As of late last week, |mach try auto| is now running in "manifest > scheduling" mode. This means the ML algorithm is running earlier in the > process and is actually creating the test chunks based on how many > important manifests there are. Therefore we are only running manifests that > are relevant to your push and are avoiding the collateral damage caused by > all the "unimportant" ones. As part of this change I've tweaked the minimum > importance threshold downward, meaning the algorithm classifies a greater > number of manifests as "important". So |mach try auto| pushes will schedule > a greater number of relevant manifests, while simultaneously scheduling > fewer manifests over all. > > One thing to note is that the chunks in your |mach try auto| pushes will > not bear any resemblance to the chunks that are created on autoland. In > fact, they may bear no resemblance to the previous |mach try auto| push you > made, even with similar changes. If you need to figure out where a given > test ran you can use Treeherder's test path filter: > > 1. Click the filter icon near the top right in Treeherder (next to > coloured circles) > 2. Select "test path" from the drop down > 3. Type out the test path you are looking for > 4. Click "Add" > > Please note that reftest isn't supported in this filter yet, and WPT > requires test identifiers (e.g `/foo` instead of > `testing/web-platform/tests/foo`). > > *What's Next?* > > There is a lot of work still coming down the pipeline. First and foremost, > we want to enable "manifest scheduling" on autoland in addition to |mach > try auto|. There are certain problems that exist on autoland (like sheriff > backfilling) that we didn't need to worry about on try. Next we'll support > manifest scheduling on all test suites (the major ones that are missing > include reftest and derivatives, WPT reftests and mochitest-webgl). Last > but not least, we're going to continue working through all of the feedback > we've been gathering over the past weeks. Thanks to everyone who has left > feedback around |mach try auto|, it's been invaluable! > > In case this post prompts you to try it out (or to revisit it again), then > please file bugs and suggestions under `Firefox Build System :: Try`, or > ping us in #test-selection on Matrix. > > Cheers! > _______________________________________________ > firefox-dev mailing list > firefox-...@mozilla.org > https://mail.mozilla.org/listinfo/firefox-dev > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
