All,
Progress on the initiative:
* After a round of hit-and-trial and exploring options, the sonarcloud +
bot is configured for apache/cloudstack repo. The quality builds are done using
Github actions [1].
* Commits pushed to main branch will get static analysis and coverage
statistics: https://sonarcloud.io/summary/overall?id=apache_cloudstack
* All new PRs opened against main branch will get static analysis, code
coverage stats as seen on some new PRs:
https://sonarcloud.io/project/pull_requests_list?id=apache_cloudstack
* Anyone can generate coverage/quality reports using "-P quality" passed to
maven, for example:
mvn -P quality -Dnoredist -Dsimulator clean install clover:aggregate
clover:clover
This will generate html output at client/target/site/ directory
* The "@acs-robot" bot makes comments such as kicks UI build if PR has UI
changes (currently pkging isn't done when java changes are detected)
* The "boring-cyborg" bot adds labels to PR based on file/paths and changes
introduced by the PR
The sonarcloud analysis on per-PR basis helps PR authors learn more about the
PR changes in an automated rule-based manner:
* robustness/reliability of code (java issues such as potential NPE, memory
and file leaks etc)
* vulnerabilities and security hotspots (code causing security issues or
can potentially lead to vulnerability)
* code smell - rule-based maintenance/debt issues (for example, complex
java code)
* unit-test coverage percentage (of the PR and also the effective % post
merging the PR)
* code duplication
When the built-in rules of sonarcloud for java code isn't honour it will show
the PR analysis as failed (for ex. by default 80% is limit on coverage). Some
of the rules/issues may be non-issues which can be marked as closed on
sonarcloud or the rules used for analysis can be changed (for ex.
include/excluse files, paths etc). I also welcome if any PMCs and committers
want to join the sonarcloud project for this access (currently I'm the admin
per https://issues.apache.org/jira/browse/INFRA-23223).
For example, here's a random PR where the bot detected 69% coverage as the PR
also adds unit tests: (personally I like PRs like this that add unit tests and
help increase overall code coverage of the codebase)
https://github.com/apache/cloudstack/pull/6358#issuecomment-1117848257
(https://sonarcloud.io/summary/new_code?id=apache_cloudstack&pullRequest=6358)
All, pl share your views:
* if we're happy with the default thresholds of sonarcloud, what rules may
be changed
* should code coverage be made a requirement (after agreeing on some min. %
coverage) at least for new features if not bugfix/small changes
Other things to explore and investigate:
* Explore and measure coverage of smoketests
* CI/CD improvements:
* Build/rekick failed Travis smoketests cases/jobs
* Speedup smoketests
* Explore weekly run of component tests
* (semi-)automated upgrade tests (may assist for release efforts)
* PR/participation changes:
* Assign reviewers based on sub-system/codebase experts based on
file/paths
* Assign PR labels by size changes
* Explore triaging of Github issues (labels, milestone, assignees etc)
[1] https://github.com/apache/cloudstack/tree/main/.github/workflows
Regards.
From: Rohit Yadav
Sent: Monday, April 18, 2022 16:12
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Community CI/CD/Triage Bot
All,
I've some updates on the initiative:
* I've added some initial support of @acs-robot which along with Github
actions is used to (a) detect UI changes and kick UI QA builds, (b) detect
java/xml changes and kick quality check and report coverage report on the PR
* Enabled coverage/quality, example the coverage reports can be seen here:
http://qa.cloudstack.cloud/coverage/ (I'll set up a daily job, on main branch)
* Quality metrics/analysis re-setup with Sonar:
https://sonarcloud.io/project/overview?id=apachecloudstack (will set up a daily
job on main branch)
To explore and investigate: (based on feedback)
* Build/re-kick failed Travis jobs
* Assign (latest) milestones to new issues and PRs
* Detect if PR needs packaging and running smoketests, build/re-kick failed
pkg jobs
* Assign PR approvers based on file-path matching
* Explore automatic triaging of issues (label, assign to maintainers)
* Label PRs by size of the PR changes (similar to k8s, s/m/l/xl/xxl/...)
* Per PR Sonar-cloud integration (not sure if this is necessary unless
contributors want this)
Please do share your ideas and thoughts. Thanks.
Regards.
From: Rohit Yadav
Sent: Thursday, April 7, 2022 13:37
To: dev@cloudstack.apache.org
Subject: Re: [DISCUSS] Community CI/CD/Triage Bot
Thanks for the feedback and suggestions Nicolas.
I've been exploring the ASF infra w
