Nicolò, Great stuff indeed. Some comments inline below Thanks
Il Mer 16 Mar 2022, 16:45 Nicolò Boschi <boschi1...@gmail.com> ha scritto: > Dear community, > > Currently we don't have an aggregate code coverage report. There's a Jacoco > configuration to generate the report after unit tests but it is not > easily readable and understandable. > > Having a better system would bring a couple of interesting aspects: > 1. We can block a pull request if the code coverage decreases - that means > there are no tests for the new code. > I am afraid that this would lead to some automatic excessive nitpicking. Let's keep this in the 'future works' basket 2. We can have an overview for each modules and discover which features are > less covered and add new tests > > I would like to introduce a new tool that reads, aggregates, stores and > reports the code coverage for the whole codebase. This tool is Codecov ( > https://about.codecov.io/) that is one of the most popular tools for open > source projects. > It brings multiple advantages: > 1. It aggregates all the Jacoco reports in a single report > 2. It keeps the coverage history > 3. It is well integrated with Github > > The idea is the following: > - Add this profiling in the CI - for each test suite and upload the results > to codecov.io > - Enable the codecov report that add a comment in the pull > > > Implementation pull: https://github.com/apache/pulsar/pull/14600 > Codecov comment in the PR: > https://github.com/apache/pulsar/pull/14600#issuecomment-1061636495 > Codecov full report: > > https://codecov.io/gh/apache/pulsar/tree/0fa212d40b2bf166b20f3155c40fdb9624f577ed > > Initially there were some doubts around whether the jacoco agent would have > added a significant overhead (execution time + memory usage). I've seen > (and you can see the checks times in my pull) that it is not relevant in > the Pulsar codebase. > > The pull only adds Jacoco coverage for unit tests. Integrating jacoco with > the integration tests (inside docker) will be more complex but it is > possible. We can do it in a second moment. > You basically need to add the jacoco agent inside the docker container, > enabling it in the java command and downloading the report files before the > container stops. > > > Note: Codecov is free for open source projects ( > https://about.codecov.io/pricing/). It is required to install the app on > Github Marketplace (https://github.com/marketplace/codecov). > Is it something we should ask to ASF infra probably. The PMC can deal with this Enrico > BR, > Nicolò Boschi >
