Lari, Thanks for putting this list together. This data really helps to prioritize which tests need to be addressed first. I'll try to find a test I can fix.
Thanks, Michael Marshall On Tue, Mar 16, 2021 at 9:27 AM Lari Hotari <lari.hot...@sagire.fi> wrote: > Dear Pulsar community members, > > Here's a report of the flaky tests in Pulsar CI during the observation > period of 2021-02-23 - 2021-03-16 > > > https://docs.google.com/spreadsheets/d/106QKMFfdLa-VSFGlggs35eMw-ZnjMUjMhA3KVBjzHLI/edit?usp=sharing > > subset of the report, 20 most flaky tests (some are already fixed): > Test method name Number of build failures due to this test > org.apache.pulsar.broker.auth.AuthorizationTest.simple 65 > > org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_5.testResetCursorCompatibility > 43 > > org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testJavaExclamationFunction > 38 > > org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testAvroSchemaFunction > 36 > > org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_4.testResetCursorCompatibility > 27 > > org.apache.pulsar.tests.integration.functions.PulsarFunctionsProcessTest.testJavaExclamationTopicPatternFunction > 25 > org.apache.pulsar.client.api.DeadLetterTopicTest.testDeadLetterTopic 20 > org.apache.pulsar.broker.service.ReplicatorTest.testConfigChange 17 > org.apache.pulsar.io.rabbitmq.sink.RabbitMQSinkTest.TestOpenAndWriteSink 17 > org.apache.pulsar.broker.service.ReplicatorTest.testResetCursorNotFail 16 > > org.apache.pulsar.broker.transaction.buffer.TransactionLowWaterMarkTest.testPendingAckLowWaterMark > 14 > > org.apache.pulsar.tests.integration.functions.PulsarFunctionsThreadTest.testAvroSchemaFunction > 13 > > org.apache.pulsar.broker.transaction.pendingack.PendingAckInMemoryDeleteTest.txnAckTestNoBatchAndSharedSubMemoryDeleteTest > 11 > org.apache.pulsar.tests.integration.functions.PulsarStateTest.testSinkState > 11 > > org.apache.pulsar.tests.integration.backwardscompatibility.ClientTest2_3.testResetCursorCompatibility > 11 > > org.apache.pulsar.broker.stats.ManagedCursorMetricsTest.testManagedCursorMetrics > 10 > > org.apache.pulsar.client.impl.MultiTopicsConsumerImplTest.testConsumerCleanupOnSubscribeFailure > 9 > > org.apache.pulsar.tests.integration.cli.CLITest.testCreateSubscriptionCommand > 9 > > org.apache.pulsar.broker.service.ReplicatorTest.testTopicReplicatedAndProducerCreate > 9 > > 4 Top flaky issues to fix: > > 1) Flaky-test: DeadLetterTopicTest.testDeadLetterTopic > https://github.com/apache/pulsar/issues/9916 > > 2) Flaky-test: ClientTestBase.resetCursorCompatibility / > ClientTest2_x.testResetCursorCompatibility > https://github.com/apache/pulsar/issues/9914 > > 3) Flaky-test: PulsarFunctionsTest.testExclamationFunction remains flaky > after fixes > https://github.com/apache/pulsar/issues/9915 > > 4) Flaky-test: PulsarFunctionsProcessTest.testAvroSchemaFunction > https://github.com/apache/pulsar/issues/9453 > > More flaky test issues: > > https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue > > Great progress has been made and our CI is more stable than before since a > lot of flaky tests have been fixed. > More contributions are welcome! Please keep up the good work! > > BR, Lari >
