A big thank you to Mick for leading the effort. This has already made a huge difference for contributors, myself included, and is a great step forward for the community. Grateful for the collaboration!
Thanks Himanshu From: Dmitry Konstantinov <netud...@gmail.com> Reply-To: "dev@cassandra.apache.org" <dev@cassandra.apache.org> Date: Thursday, August 28, 2025 at 6:05 AM To: "dev@cassandra.apache.org" <dev@cassandra.apache.org> Subject: RE: [EXTERNAL] [ANNOUNCE] Pre-commit CI @ pre-ci.cassandra.apache.org , an AWS donation CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. It’s a huge milestone! Many thanks to Mick and everyone else involved in setting up and configuring the CI. Thanks also to Amazon and the other companies that donated hardware resources to keep the infrastructure running. This is especially helpful for committers who don’t have the resources to run their own CI system, like me, and it makes the contribution process more efficient and pleasant. Hopefully, this will lead to more contributions and contributors. On Wed, 27 Aug 2025 at 22:50, Mick <m...@apache.org<mailto:m...@apache.org>> wrote: tl;dr I am super excited to share with you all the availability of https://pre-ci.cassandra.apache.org , made possible from a very generous donation by Amazon. Using AWS donated credits we now have a fast open-source CI system for pre-commit testing. This is intended primarily for committers (and trusted contributors) that do not have access to a company-private CI system. Contact me off-list if that's you. … This huge donation from Amazon and builds on years of work on our CI and testing scripts, accumulating in CASSANDRA-18145¹. The project now has an AWS account (controlled by ASF Infra), and into it Amazon has provided a significant amount of credits for the purpose of our pre-commit testing. In the C* project pre-commit CI has been a major hurdle and blocker to new contributors, unaffiliated committers, and generally getting patches committed. Too much of our momentum has been led by contributors and committers employed in companies that can pay to run their own in-house CI systems. It is not in the slightest their fault, nor their employers' priority, but it has held the project back from its potential as well as not being in the proper spirit of an open source project and community. Amazon has recognised this and stepped forward as a true champion of our open-source community, providing both the AWS credits, as well as time in helping setting it up, testing and documenting it. The work has also involved many steps and people the past five years! since that first mention of a "Stand-alone Pipeline Run"². This work has gone from having a standalone Jenkinsfile, testing images and scripts that can be used anywhere, both pre-commit and post-commit, to a Helm chart to deploy ci-cassandra.a.o Jenkins clones in any k8s cluster of your choosing. This has also included getting our CI pipeline generating aggregated test result artefact files that can easily be attached to jira tickets, and archived permanently. These objectives you can see listed in CASSANDRA-18137³. For all this special thanks goes out to Richa Mishra, Nishant Barola, Aleksandr Volochnev, Brandon Hsieh, Himanshu Jindal, Steve Mayszak, Dmitry Konstantinov, Maxim Muzafarov, Berenguer Blasi, David Capwell, Brandon Williams, Ekaterina Dimitrova, Henrik Ingo, Shankar Ramaswamy, Josh McKenzie, Stefan Miklosovic, Gavin McDonald and no doubt others I've forgotten! Yeah, it feels like almost everyone, it has been a journey! It is not perfect: there is (always) room for improvement; and for now I think next steps should be around further time+cost optimisations and pipeline configurability. I'm hoping folk can jump in and help with these. What is provided today is a way for everyone anywhere to immutably provision a ci-cassandra.a.o clone on k8s. You avoid any need to install, know or operate Jenkins (you do not even have to ever once open the Jenkins UI – see below) and you do not need to wait more than a few hours to run the "post-commit" pipeline with 140-200k tests. The correlation to pipeline time is now how many nodes your k8s can scale to (ofc limitations exist, but those fine details are other convo), and with the cluster scaling down to one node you get a cost-effective pay-what-you-use deployment. https://pre-ci.cassandra.apache.org runs on EKS, and another instance for DataStax employees ( http://ci-cassandra.infra.datastax.com ) has been in use on GKE for over a year now. We have a CI system that is relatively ops-free, has the same turnaround time as CircleCI and costs over an order of magnitude less. This has been designed for downstream C* products to be able to re-use, and indeed it is a setup that any project can easily use. The setup introduces repeatability in how we test. The Jenkinsfile, the docker images and the test scripts, are the same everywhere, including locally. Reproducing obscure failures in post-commit should be easier. If you would like to set up your own ci-cassandra clone, the docs for both GKE are found in-tree under .jenkins/k8s/ ,and the docs for EKS coming soon. There is a command line interface that can be used for both provisioning as well as everyday CI use. Some examples: ``` # Install the ci-cassandra helm chat in your current k8s context, creating the ci-cassandra clone .build/run-ci --only-setup # Run your current branch through CI with the skinny profile. # Waits til completion, and creates ci_summary*.html & results_details*.tar.xz files to attach to jira .build/run-ci --profile skinny # If you don't have kubectl access to the cluster, just specify the jenkins url and user .build/run-ci --profile skinny --url pre-ci.cassandra.apache.org<http://pre-ci.cassandra.apache.org> --user myuser ``` Huge huge praise and thanks to Amazon here: they really deserve it, they have taken this over the finish line; but the opportunity deserves to repeat all our gratitudes to all the very significant ci-cassandra.apache.org<http://ci-cassandra.apache.org> donations that help us keep the project running: NetApp, DataStax, Huawei and iland. ¹) CASSANDRA-18145 – Run entire Cassandra Jenkins in any k8s (immutable provisioning) ²) https://lists.apache.org/thread/ch768l8y95npno9o9y130fjob1h675z8 ³) CASSANDRA-18137 – Repeatable ci-cassandra.a.o (Epic) -- Dmitry Konstantinov
