I'd like to give you a quick update on the work that has been done lately on running tests using CircleCI. Please let me know if you have any objections or don't think this is going into the right direction, or have any other feedback!
We've been using CircleCI for a while now and results are used on constant basis for new patches. Not only by committers, but also by casual contributors to run unit tests. Looks like people find the service valuable and we should keep using it. Therefor I'd like to make some improvements that will make it easier to add new tests and to continue making CircleCI an option for all contributors, both on paid and free plans. The general idea of the changes implemented in #14806, is to consolidate the existing config to make it more modular and have smaller jobs that can be scheduled ad-hoc by the developer, instead of running a few big jobs on every commit. Reorganizing and breaking up the existing config was done using the new 2.1 config features. Starting jobs on request, instead of automatically, is done using the manual approval feature, i.e. you now have to click on that job in the workflow page in order to start it. I'd like to see us having smaller, more specialized groups of tests that we can run more selectively during development, while still being able to run bigger tests before committing, or firing up all of them during testing and releasing. Other example of smaller jobs would be testing coverage (#14788) or cqlsh tests (#14298). But also individual jobs for different ant targets, like burn, stress or benchmarks. We'd now also be able to run tests using different docker images and different JDKs. I've already updated the used image to also include Java 11 and added unit and dtest jobs to the config for that. It's now really easy to run tests on Java 11, although these won't pass yet. It seems to be important to me to have this kind of flexibility, given the increasingly diverse ecosystem of Java distributions. We can also add jobs for packaging and doing smoke tests by installing and starting packages on different docker images (Redhat, Debian, Ubuntu,..) at a later point. As for the paid vs free plans issue, I'd also like us to discuss how we can make tests faster and less resource intensive in general. As a desired consequence, we'd be able to move away from multi-node dtests, to something that can be run using the free plan. I'm looking forward to see if #14821 can get us into that direction. Ideally we can add these tests into a job that can be completed on the free plan and encourage contributors to add new tests there, instead of having to write a dtest, which they won't be able to run on CircleCI without a paid plan. Whats changing for you as a CircleCI user? * All tests, except unit tests, will need to be started manually and will not run on every commit (this can be further discussed and changed anytime if needed) * Updating the config.yml file now requires using the CircleCI cli tool and should not be done directly (see #14806 for technical details) * High resource settings can be enabled using a script/patch, either run manually or as commit hook (again see ticket for details) * Both free and paid plan users now have more tests to run As already mentioned, please let me know if you have any thoughts on this, or if you think this is going into the wrong direction. Thanks. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
