Jun, 1) The reports mentioned in the KIP will need to be built. As a start, I think we can use a cron-based GitHub Action that produces a markdown report. Longer term, we can maybe look into some static site generator like GitHub Pages for hosting weekly reports.
2) In my opinion, any test which could possibly be flaky should be an integration test. For example, any test using a real network, system time, threads, etc, should be marked as an integration test. -David On Mon, Sep 30, 2024 at 12:28 PM Jun Rao <j...@confluent.io.invalid> wrote: > Hi, David, > > Thanks for the report. > > 1. Where could we see the reports (quarantined tests, etc) mentioned in the > KIP? > 2. The quarantine process seems to only apply to integration tests. What's > our recommendation for flaky unit tests? > > Jun > > On Thu, Sep 26, 2024 at 1:34 PM David Arthur <mum...@gmail.com> wrote: > > > If there is no more feedback on this, I'll go ahead and move to a vote. > > > > -David > > > > On Sun, Sep 22, 2024 at 11:04 AM Chia-Ping Tsai <chia7...@gmail.com> > > wrote: > > > > > > > > > > > > David Arthur <mum...@gmail.com> 於 2024年9月22日 晚上10:07 寫道： > > > > > > > > Q2: Yes, I think we should run the quarantined tests on all CI > builds, > > > PRs > > > > and trunk. We can achieve this with --rerun-tasks. This will let PR > > > authors > > > > gain feedback about their changes affect on the flaky tests. We could > > > even > > > > create a PR-specific report that shows if their changes improved or > > > > worsened the flakiness of the quarantined tests. > > > > > > I guess it will be an individual status like “Gradle Build Scan”? The > > > failure of quarantined tests does not obstruct us from merging PR > unless > > > the target of PR is to fix specific flaky. > > > > > > Thanks, > > > Chia-Ping > > > > > > > > -- > > David Arthur > > > -- David Arthur
