On Fri, 27 Jan 2023 at 13:11, Peter Maydell <peter.mayd...@linaro.org> wrote: > > On Fri, 27 Jan 2023 at 12:39, Kevin Wolf <kw...@redhat.com> wrote: > > > > Am 26.01.2023 um 15:28 hat Peter Maydell geschrieben: > > > On Thu, 26 Jan 2023 at 14:25, Stefan Hajnoczi <stefa...@gmail.com> wrote: > > > > > > > > Are you batching pull requests? I used that approach last release > > > > cycle. CI takes so long to run that I didn't want to run it for every > > > > pull request. Batching worked well overall. > > > > > > No, I just do one test per pullreq. IME the CI is flaky > > > enough that I don't really want to batch it up, and it > > > isn't so slow that I build up a backlog of unprocessed > > > requests. > > > > But obviously so slow that we've run out of minutes. It would be good if > > this didn't happen every month in the future. > > > > If it worked well enough for Stefan, I think it would be worth trying to > > batch some pull requests going forward. What is the downside of it? If > > CI fails and flaky tests seem to be at fault, I assume you just re-run > > the job, no matter whether it tests a single pull request or two or > > three of them? > > It means that if something fails it's harder to see whether > it was pullreq A or pullreq B. It also means there's a higher > cost to "abandon processing the merge and try a different one > to see if that one goes through and come back to this one later", > which is also something I sometimes do in an attempt to figure > out whether a problem is the usual flaky CI or not.
Put another way, I think that an important thing we need to do is to look at all the CI failures we get and track down exactly why we have a bunch of intermittent failures and squash them. A lot of these problems are secondary things that we've ended up with because we have a lot of flakiness. -- PMM
