Hi all,
There's an improved code coverage collection solution in place for the
apache/pulsar master branch. In the past, there were several problems
with the code coverage collection and reporting solution:
- Codecov results were unreliable because of failed uploads
- The Codecov comments on PRs were reporting false numbers because of
unreliable ways to collect code coverage.
- Code coverage didn't track any classes that were outside of the
project where the tests resided.
- Code coverage didn't track integration or system tests that are run as
part of the CI pipeline.
These problems have been addressed with multiple pull requests during
this week. Please ask if you are interested in the details.
You can go to https://app.codecov.io/gh/apache/pulsar to see the latest
code coverage report for the master branch.
I hope this information will help design better tests for critical areas
of Pulsar. Codecov pull request comments will be more useful for future
PRs when the code coverage information is accurate and consistent. It
will also help track progress over time.
Since this is related to improving tests, I'd like to remind that we
have Gradle Enterprise in use and it contains build scans for scheduled
master branch builds. Gradle Enterprise contains useful analysis for
tests, for example the long running test report could be a starting
point for reducing build times:
https://ge.apache.org/scans/tests?search.rootProjectNames=Pulsar&tests.sortField=MEAN_DURATION
There's a very detailed breakdown of test classes, for example
https://ge.apache.org/scans/tests?search.rootProjectNames=Pulsar&tests.container=org.apache.pulsar.broker.admin.TopicPoliciesTest&tests.sortField=MEAN_DURATION
I hope this helps planning test improvements. I'm looking forward to your
contributions to improve Pulsar tests. :)
-Lari
