Dear Pulsar community members, In order to improve our CI, we will have to fix the flaky tests. In some cases it might be necessary to replace an existing test with a redesigned test.
The draft PIP "Changes to flaky test handling" document <https://docs.google.com/document/d/10lmn4pW1IsT_8D1ZE0vMjASX0HhjdGdjB794iyScwns/edit?usp=sharing> lists the top 10 flaky tests. A lot of them have already been address by pull requests in the past week or so. This is the list of recent PRs that fix flaky tests from the top 10 flaky tests list: https://github.com/apache/pulsar/pull/9286 https://github.com/apache/pulsar/pull/9243 https://github.com/apache/pulsar/pull/9258 https://github.com/apache/pulsar/pull/9356 These are the GH issues for the remaining ones in the top 10 flaky tests list: https://github.com/apache/pulsar/issues/6368 https://github.com/apache/pulsar/issues/9369 https://github.com/apache/pulsar/issues/9368 If you would like to help to fix flaky tests you can pick one of the open issues above. Just add a comment on the issue when you start working on it so that we can coordinate activities. It is also helpful to report a flaky test when you encounter one. I've been using this type of template for reporting a flaky test: https://gist.github.com/lhotari/a5c67359b362b4f3d8729330d65a2298 . The issues #9368 and #9369 have been reported using this template. Search for the test name before reporting so that we don't end up with duplicates. The issues #6368, #9369 and #9368 are the 3 next important issues to fix. I'm planning to create a more extensive list of the flaky failures so that we can target the most flaky ones when we continue fixing the flaky tests. I have some scripts in development to assist in mining the Pulsar Github Action workflow run logs. This is a search to find flaky issues in Pulsar GH issues: https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen Looking forward to the contributions for fixing flaky tests, BR, Lari
