Fantastic initiative.
On Thu, Jul 21, 2022 at 10:59 PM Nicolò Boschi <boschi1...@gmail.com> wrote: > Dear community, > > Here's a report of the flaky tests in Pulsar CI during the observation > period of 2022-07-12 - 2022-07-19 > > > https://docs.google.com/spreadsheets/d/165FHpHjs5fHccSsmQM4beeg6brn-zfUjcrXf6xAu4yQ/edit?usp=sharing > > The report contains a subset of the test failures. > The flaky tests are observed from builds of merged PRs. > The GitHub Actions logs will be checked for builds where the SHA of the > head of the PR matches the SHA which got merged. > This ensures that all found exceptions are real flakes, since no changes > were made to the PR to make the tests pass later > so that the PR was merged successfully. > > > > Top 5 flaky issues to fix: > > org.apache.pulsar.tests.integration.functions.java.PulsarFunctionsJavaThreadTest.testJavaLoggingFunction > 92 > org.apache.pulsar.functions.worker.PulsarFunctionTlsTest.tearDown 36 > > org.apache.pulsar.broker.service.persistent.PersistentSubscriptionMessageDispatchStreamingDispatcherThrottlingTest.testDispatchRateCompatibility2 > 27 > > org.apache.pulsar.broker.service.persistent.PersistentDispatcherFailoverConsumerStreamingDispatcherTest.testAddRemoveConsumer > 23 > > org.apache.pulsar.broker.service.PersistentDispatcherFailoverConsumerTest.testAddRemoveConsumer > 22 > > > Markdown formatted summary reports for each test class can be accessed at > > https://github.com/nicoloboschi/pulsar-flakes/tree/master/2022-07-12-to-2022-07-19 > > > More flaky test issues: > > https://github.com/apache/pulsar/issues?q=flaky+sort%3Aupdated-desc+is%3Aopen+is:issue > > We need more help in addressing the flaky tests. Please join the efforts > so that we can get CI to a more stable state. > > To coordinate the work, > 1) please search for an existing issues or search for all flaky issues with > "flaky" or the test class name (without package) in the search: > https://github.com/apache/pulsar/issues?q=is%3Aopen+flaky > +sort%3Aupdated-desc > 2) If there isn't an issue for a particular flaky test failure that you'd > like to fix, please create an issue using the "Flaky test" template at > https://github.com/apache/pulsar/issues/new/choose > 3) Please comment on the issue that you are working on. > > We have a few active contributors working on the flaky tests, thanks for > the contributions. > > I'm looking forward to more contributors joining the efforts. Please join > the #testing channel on Slack if you'd like to ask questions and tips about > reproducing flaky tests locally and how to fix them. > Sharing stories about fixing flaky tests is also helpful for sharing the > knowledge about how flaky tests can be fixed. That's also a valuable way to > contribute. > Some flaky tests might be actual real production code bugs. Fixing > the flaky test might result in fixing a real production code bug. > > More contributions are welcome! Please keep up the good work! > > Thanks, > Nicolò Boschi >
