Q0: > However, the next description: “ Placing a new test into the quarantine will be made at the discretion of the PR authors and/or committers. It should not be compulsory.”
Thanks, this was left over from the first iteration of the design, I'll remove this. Since the process is automated for new integration tests, it seems fine to do it for all new integration tests (rather than leaving it up to the author). Q1: the only way a developer can "place a test into quarantine" is by explicitly adding the annotation Q2: Yes, I think we should run the quarantined tests on all CI builds, PRs and trunk. We can achieve this with --rerun-tasks. This will let PR authors gain feedback about their changes affect on the flaky tests. We could even create a PR-specific report that shows if their changes improved or worsened the flakiness of the quarantined tests. -David On Sat, Sep 21, 2024 at 11:49 PM Chia-Ping Tsai <chia7...@gmail.com> wrote: > hi David > > Thanks for all response. I have a couple of questions about the quarantine > test and PR flow > > > Q0: > > I am still a bit confused about “should we put new integration tests into > quarantine manually/automatically?” To quote the KIP: “ Automatically > placing a new test into the quarantine will allow us to observe its > behavior without the risk of failing builds. After a few days of successful > runs, these tests will graduate from the quarantine and be run as part of > the main suite. ” > > It seems those new tests will be placed into quarantine automatically. > > However, the next description: “ Placing a new test into the quarantine > will be made at the discretion of the PR authors and/or committers. It > should not be compulsory.” > > It seems it is up to authors and/or committers. > > > Q1: > > Could you please share more details about “ Placing a new test into the > quarantine”? Do you mean “add the flaky annotation to the new integration > tests in the PR”? > > > Q2: > > Are the quarantine tests included by PR’s CI? If yes, could we (committers > and author) ignore the failure of quarantine tests? If not, should we > remove the retry from PR flow as the non-quarantine tests should be stable? > > Best, > Chia-Ping -- David Arthur
