Update regarding Codecov improvements for apache/pulsar CI: - fixed issue with Jacoco coverage data not getting stored in files: https://github.com/apache/pulsar/pull/19947 This seemed to be a broader issues since the reported total code coverage increased to about 72.8% with this fix, example https://app.codecov.io/gh/apache/pulsar/pull/19947/tree .
There's a workaround for Codecov upload issue in progress. More details in the comment https://github.com/apache/pulsar/issues/19952#issuecomment-1487997039 . This is waiting for ASF Infra to resolve https://issues.apache.org/jira/browse/INFRA-24399 . After this, I believe that Codecov will be reasonably stable in our CI. Actions will be needed for individuals for adding a Codecov upload token for builds in personal forks. I'll add instructions for that while resolving #19952. -Lari On 2023/03/24 09:51:51 Lari Hotari wrote: > Thanks for sharing the pain. That's the first step in improving something > that is painful. > > For the flaky tests GitHub Actions workflow pulsar-ci-flaky.yaml, the Codecov > upload should be a separate job in the workflow so that the upload could be > retried separately without running all tests. This type of approach is > already used in the main GitHub Actions workflow, "Pulsar CI". > Contributions are welcome! > > We could also consider disabling codecov for pull request builds until > someone who cares about test code coverage metrics picks up the work. > > Code coverage is the first metric that most will ask about tests. It's not > the only metric that matter, but it is something that helps understand what > parts of the code isn't even run in our tests. It will also help plan > improvements to tests. > > Codecov upload fails very frequently with errors such as > https://github.com/codecov/codecov-action/issues/837 and > https://github.com/codecov/codecov-action/issues/598 > One possible resolution is > https://community.codecov.com/t/upload-issues-unable-to-locate-build-via-github-actions-api/3954 > . > It's possible to make the codecov upload more stable by providing a token. > This should be done for the master branch build so that the baseline code > coverage metrics would succeed. For pull requests, the solution is to make > the codecov upload retryable also in pulsar-ci-flaky.yaml. In addition, it > could be made optional for builds in own forks. > > We should find a way as a development community to get code coverage metrics > solution working. It is valuable even if an individual developer doesn't care > about it at the moment. > We need more Pulsar contributors to stand up that care about the quality > aspects of our code base. Any volunteers? > > -Lari > > On 2023/03/21 10:50:17 tison wrote: > > For example > > https://github.com/apache/pulsar/actions/runs/4454158774/jobs/7867745340?pr=19842 > > > > I'm wondering if anyone cares about the report and if it helps you during > > the coding or reviewing process? Now it generates a few of noise but I just > > omit the report it gives ;-) > > > > For the issue itself, it seems some artifacts don't retain properly. > > > > Best, > > tison. > > >
