On Wed, May 6, 2020 at 4:04 PM Steve Fink <sf...@mozilla.com> wrote: > Would it work to make a `mach try fuzzy --auto` or similar entrypoint, > where it brings up the fzf selector with the automatic set of tests as a > starting point? >
Yes, this is something I've thought about as well. We could remove chunk numbers from the fzf interface so you are basically selecting configurations rather than tasks. Then ML would supply the tests to run and the taskgraph would ensure only the cross section of tests + configurations end up being scheduled. The same approach could apply to `./mach try chooser`. I filed bug 1635921 <https://bugzilla.mozilla.org/show_bug.cgi?id=1635921> to keep track of this idea. Thanks! On 5/6/20 6:55 AM, Andrew Halberstadt wrote: > > Hello everyone, > > > > A handful of us have been hard at work on an initiative to become smarter > > about which tests we schedule for a given push. We are approaching our > > first major milestone, a try selector that can automatically pick which > > tests are most relevant to your push on your behalf. You can use it today > > by pulling the latest central, then running: > > > > $ ./mach try auto > > > > Running this command will feed your patch(es) through a machine learning > > algorithm that will guess which tests are most likely to be relevant. > While > > early results look promising, the tool is considered very experimental. > So > > we'd love to collect feedback from a wider developer audience before > > recommending that people start using it by default. > > > > Feedback we are looking for includes: > > > > 1. Pushes where |mach try auto| failed to detect a regression (you either > > got backed out or detected it yourself via some other means). > > 2. Sentiment on resource usage. Does |mach try auto| schedule too much or > > too little? How often are you willing to tolerate a backout? > > 3. Bug reports, especially decision task failures. > > 4. Feature requests. What would you find most useful when using this > tool? > > For example, a flag to increase/decrease the number of tasks that get > run. > > Or maybe a way to explicitly choose which platforms/configurations to run > > on. > > > > If you have any questions or would like to provide feedback, please reach > > out to us in #test-selection:mozilla.org > > < > https://matrix.to/#/!ahqVzULkEEXTziqCfU:mozilla.org?via=mozilla.org&via=t2bot.io > > > > on Matrix. You can also file bug and feature reports in Firefox Build > > System :: Try > > < > https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Build%20System&component=Try > > > > . > > > > Ongoing improvements to the algorithm combined with your feedback and an > > ever growing dataset to train the ML, means that |mach try auto| will > > continuously improve over time. Eventually we'll recommend everyone use > it > > by default as well as use the same algorithm on autoland. > > > > Thanks in advance for your help in getting us there! > > -Andrew > > > > p.s If you've previously tried |mach try auto| and dismissed it for > running > > too many tasks or frequent decision task failures, please pull and give > it > > another shot! > > _______________________________________________ > > dev-platform mailing list > > dev-platform@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-platform > > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
