Dear Pulsar community members, Here's a report of the flaky tests in Pulsar CI during the observation period of 2021-02-23 - 2021-03-16
https://docs.google.com/spreadsheets/d/106QKMFfdLa-VSFGlggs35eMw-ZnjMUjMhA3KVBjzHLI/edit?usp=sharing subset of the report, 20 most flaky tests (some are already fixed): Test method name Number of build failures due to this test org.apache.pulsar.broker.auth.AuthorizationTest.simple 65 org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_5.testResetCursorCompatibility 43 org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testJavaExclamationFunction 38 org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testAvroSchemaFunction 36 org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_4.testResetCursorCompatibility 27 org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testJavaExclamationTopicPatternFunction 25 org.apache.pulsar.client.api.DeadLetterTopicTest.testDeadLetterTopic 20 org.apache.pulsar.broker.service.ReplicatorTest.testConfigChange 17 org.apache.pulsar.io.rabbitmq.sink.RabbitMQSinkTest.TestOpenAndWriteSink 17 org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail 16 org.apache.pulsar.broker.transaction.buffer.TransactionLowWaterMarkTest.testPendingAckLowWaterMark 14 org.apache.pulsar.tests.integration.functions.PulsarFunctionsThreadTest.testAvroSchemaFunction 13 org.apache.pulsar.broker.transaction.pendingack.PendingAckInMemoryDeleteTest.txnAckTestNoBatchAndSharedSubMemoryDeleteTest 11 org.apache.pulsar.tests.integration.functions.PulsarStateTest.testSinkState 11 org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_3.testResetCursorCompatibility 11 org.apache.pulsar.broker.stats.ManagedCursorMetricsTest.testManagedCursorMetrics 10 org.apache.pulsar.client.impl.MultiTopicsConsumerImplTest.testConsumerCleanupOnSubscribeFailure 9 org.apache.pulsar.tests.integration.cli.CLITest.testCreateSubscriptionCommand 9 org.apache.pulsar.broker.service.ReplicatorTest.testTopicReplicatedAndProducerCreate 9 4 Top flaky issues to fix: 1) Flaky-test: DeadLetterTopicTest.testDeadLetterTopic https://github.com/apache/pulsar/issues/9916 2) Flaky-test: ClientTestBase.resetCursorCompatibility / ClientTest2_x.testResetCursorCompatibility https://github.com/apache/pulsar/issues/9914 3) Flaky-test: PulsarFunctionsTest.testExclamationFunction remains flaky after fixes https://github.com/apache/pulsar/issues/9915 4) Flaky-test: PulsarFunctionsProcessTest.testAvroSchemaFunction https://github.com/apache/pulsar/issues/9453 More flaky test issues: https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue Great progress has been made and our CI is more stable than before since a lot of flaky tests have been fixed. More contributions are welcome! Please keep up the good work! BR, Lari
