Dear Pulsar community members, Here's a report of the flaky tests in Pulsar CI during the observation period of 2021-03-16 - 2021-03-23 . The report has been also updated to the Google Sheet, https://docs.google.com/spreadsheets/d/106QKMFfdLa-VSFGlggs35eMw-ZnjMUjMhA3KVBjzHLI/edit#gid=944956527
Test method name Number of build failures due to this test org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testJavaExclamationTopicPatternFunction 48 org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testJavaExclamationFunction 38 org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_5.testResetCursorCompatibility 26 org.apache.pulsar.client.api.DeadLetterTopicTest.testDeadLetterTopicByCustomTopicName 18 org.apache.pulsar.broker.service.persistent.SimpleProducerConsumerTestStreamingDispatcherTest.testRedeliveryFailOverConsumer 17 org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_4.testResetCursorCompatibility 15 org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testAvroSchemaFunction 15 org.apache.pulsar.broker.service.persistent.PersistentTopicStreamingDispatcherE2ETest.testBrokerConnectionStats 11 org.apache.pulsar.client.impl.RawReaderTest.testAcknowledgeWithProperties 10 org.apache.pulsar.broker.service.ReplicatorTest.testConfigChange 10 org.apache.pulsar.tests.integration.proxy.TestProxy.testProxyRequestBodyRedirect 9 org.apache.pulsar.tests.integration.proxy.TestProxy.testProxyWithNoServiceDiscoveryProxyConnectsViaURL 9 org.apache.pulsar.tests.integration.proxy.TestProxy.testProxyWithServiceDiscovery 9 org.apache.pulsar.tests.integration.cli.CLITest.testCreateSubscriptionCommand 9 org.apache.pulsar.broker.service.PersistentTopicE2ETest.testBrokerConnectionStats 8 org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_2.testResetCursorCompatibility 7 org.apache.pulsar.broker.admin.AdminApiTest2.testTopicPoliciesWithMultiBroker 6 org.apache.pulsar.tests.integration.cli.CLITest.testSetInfiniteRetention 5 org.apache.pulsar.client.impl.MultiTopicsReaderTest.testReadMessageWithoutBatchingWithMessageInclusive 5 org.apache.pulsar.client.api.DeadLetterTopicTest.testDeadLetterTopic 4 org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testTumblingCountWindowTest 4 org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testSlidingCountWindowTest 4 org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_3.testResetCursorCompatibility 3 org.apache.pulsar.broker.service.ConsumedLedgersTrimTest.testConsumedLedgersTrimNoSubscriptions 3 observation period 2021-03-16 - 2021-03-23 Some tests have already been fixed (for example RawReaderTest). It seems that PulsarFunctionsProcessTest is now the most flaky test even though attempts have been made to fix it. It takes time until the fixes get picked up by the builds since we were previously building the PR branch HEAD commit. The fixes would arrive in PRs only after a rebase (or merge). That has been recently changed so that we build the PR branch merge commit. This change was done by PR https://github.com/apache/pulsar/pull/9963 . It will take time until PRs include that change. After that is includes, fixes to flaky tests will be delivered to all PRs after the fixes are merged to master. *We need more help in address the flaky tests. Please join the efforts so that we get over the bump. * To coordinate the work, 1) please search for an existing issues or search for all flaky issues with "flaky" or the test class name (without package) in the search: https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky+sort%3Aupdated-desc 2) If there isn't an issue for a particular flaky test failure that you'd like to fix, please create an issue using the "Flaky test" template at https://github.com/apache/pulsar/issues/new/choose 3) Please comment on the issue that you are working on it. We have a few active contributors working on the flaky tests, thanks for the contributions. I'm looking forward to more contributors joining the efforts. Please join the #testing channel on Slack if you'd like to ask questions and tips about reproducing flaky tests locally and how to fix them. Sharing stories about fixing flaky tests is also helpful for sharing the knowledge about how flaky tests can be fixed. That's a valuable way to contribute too. Current contributors, please keep up the good work! New contributors, you are welcome to join the efforts! You will learn about Pulsar and its internals as a side-effect. If you'd love to learn Pulsar internals and Pulsar OSS development, start by fixing flaky tests. BR, -Lari
