Dear Pulsar community, Here's a report of the flaky tests in Pulsar CI during the observation period of 2023-10-27 to 2023-11-06.
The Pulsar CI is in fairly good shape at the moment. We have been able to reduce flakiness, and it is not currently slowing down PR processing significantly. The flaky test reporting has highlighted these tests as the most flaky ones: https://github.com/apache/pulsar/issues/21287 PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer 11 failures https://github.com/apache/pulsar/issues/13953 PulsarDebeziumOracleSourceTest.testDebeziumOracleDbSource 11 failures https://github.com/apache/pulsar/issues/21469 fix PR: https://github.com/apache/pulsar/pull/21479 ExtensibleLoadManagerImplTest.testCheckOwnershipAsync 6 failures https://github.com/apache/pulsar/issues/16786 PulsarFunctionsJavaProcessTest.testJavaExclamationFunction 6 failures https://github.com/apache/pulsar/issues/21292 BrokerServiceLookupTest.testLookupConnectionNotCloseIfGetUnloadingExOrMetadataEx 3 failures Putting focus on fixing the most flaky tests will be helpful. More details in this Google sheet: https://docs.google.com/spreadsheets/d/1gtu-XrLumjBFPk9kDKcJOQfxsvIE2EiuZO7IB7ab6q0/edit Detailed reports and flaky test reporting source: https://github.com/lhotari/pulsar-flakes/tree/master/2023-10-27-to-2023-11-06 In addition to the flaky test reporting, there's also thread leak reporting in the Pulsar CI build. Thread leaks could be one source of test flakiness and that's why it is helpful to fix thread leaks in our tests and not introduce new thread leaks. You can view the thread leak reports in the unit test jobs in any of the most recently Pulsar CI builds. For example, the scheduled builds for the master branch can be found here: https://github.com/apache/pulsar/actions/workflows/pulsar-ci.yaml?query=branch%3Amaster+event%3Aschedule . example of a thread leak report: https://github.com/apache/pulsar/actions/runs/6784235160/job/18440670499#step:16:23 (can be viewed by clicking on the "Report detected thread leaks" in all unit test jobs) To coordinate the work of fixing flaky tests, 1) please search for an existing issues or search for all flaky issues with "flaky" or the test class name (without package) in the search: https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc 2) If there isn't an issue for a particular flaky test failure that you'd like to fix, please create an issue using the "Flaky test" template at https://github.com/apache/pulsar/issues/new/choose 3) Please comment on the issue that you are working on it. Let's continue to reduce the flakiness to make contributing to Pulsar a better experience! -Lari
