Dear Pulsar community members, Here's a report of the flaky tests in Pulsar CI during the observation period of 2021-08-25 - 2021-09-01 . The full report has been also updated to a Google Sheet, https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit#gid=0
Test method name Number of build failures due to this test org.apache.pulsar.metadata.LockManagerTest.revalidateLockOnDifferentSession 94 org.apache.pulsar.metadata.LockManagerTest.revalidateLockWithinSameSession 45 org.apache.pulsar.metadata.LockManagerTest.updateValue 45 org.apache.pulsar.metadata.ZKSessionTest.testReacquireLocksAfterSessionLost 29 org.apache.pulsar.broker.stats.ManagedCursorMetricsTest.testCursorReadWriteMetrics 23 org.apache.pulsar.client.api.ClientErrorsTest.testProducerReconnect 20 org.apache.pulsar.client.api.SimpleProducerConsumerTest.testRedeliveryOfFailedMessages 19 org.apache.pulsar.client.api.DeadLetterTopicTest.testDeadLetterTopic 19 org.apache.pulsar.client.api.ClientDeduplicationFailureTest.testClientDeduplicationCorrectnessWithFailure 19 org.apache.pulsar.client.api.SimpleProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect 19 org.apache.pulsar.client.api.ClientDeduplicationFailureTest.testClientDeduplicationWithBkFailure 19 org.apache.pulsar.client.api.DeadLetterTopicTest.testDeadLetterTopicByCustomTopicName 19 org.apache.pulsar.client.api.SimpleProducerConsumerTest.testRedeliveryFailOverConsumer 19 org.apache.pulsar.client.impl.SequenceIdWithErrorTest.testCheckSequenceId 19 org.apache.pulsar.broker.stats.ManagedCursorMetricsTest.testManagedCursorMetrics 19 org.apache.pulsar.client.api.v1.V1_ProducerConsumerTest.testActiveAndInActiveConsumerEntryCacheBehavior 18 org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testAsyncBatchReceivePartitionedTopic 18 org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testBatchReceiveNonPartitionedTopic 18 org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testBatchReceiveAndRedeliveryPartitionedTopic 18 org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testBatchReceivePartitionedTopic 18 org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testBatchReceiveAndRedeliveryNonPartitionedTopic 18 org.apache.pulsar.client.api.ConsumerBatchReceiveTest.testAsyncBatchReceiveNonPartitionedTopic 18 org.apache.pulsar.client.api.ConsumerBatchReceiveTest.verifyBatchSizeIsEqualToPolicyConfiguration 18 org.apache.pulsar.broker.stats.PrometheusMetricsTest.testDuplicateMetricTypeDefinitions 16 org.apache.pulsar.client.api.KeySharedSubscriptionTest.testRemoveFirstConsumer 12 observation period 2021-08-25 to 2021-09-01 Markdown formatted summary reports for each test class can be accessed at https://github.com/lhotari/pulsar-flakes/tree/master/2021-08-25-to-2021-09-01/1630481590725 (type 't' to activate search by filename in GitHub UI, then type the test class name to find the report) *We need more help in addressing the flaky tests. Please join the efforts so that we can get CI to a more stable state. * To coordinate the work, 1) please search for an existing issues or search for all flaky issues with "flaky" or the test class name (without package) in the search: https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky +sort%3Aupdated-desc 2) If there isn't an issue for a particular flaky test failure that you'd like to fix, please create an issue using the "Flaky test" template at https://github.com/apache/pulsar/issues/new/choose 3) Please comment on the issue that you are working on it. We have a few active contributors working on the flaky tests, thanks for the contributions. I'm looking forward to more contributors joining the efforts. Please join the #testing channel on Slack if you'd like to ask questions and tips about reproducing flaky tests locally and how to fix them. Sharing stories about fixing flaky tests is also helpful for sharing the knowledge about how flaky tests can be fixed. That's a valuable way to contribute too. *Current contributors, please keep up the good work! * *New contributors, you are welcome to join the efforts! You will learn about Pulsar and its internals as a side-effect. * *If you'd love to learn Pulsar internals and Pulsar OSS development, start by fixing flaky tests.* BR, -Lari
