It’s a huge milestone! Many thanks to Mick and everyone else involved in setting up and configuring the CI. Thanks also to Amazon and the other companies that donated hardware resources to keep the infrastructure running.
This is especially helpful for committers who don’t have the resources to run their own CI system, like me, and it makes the contribution process more efficient and pleasant. Hopefully, this will lead to more contributions and contributors. On Wed, 27 Aug 2025 at 22:50, Mick <m...@apache.org> wrote: > > tl;dr > I am super excited to share with you all the availability of > https://pre-ci.cassandra.apache.org , made possible from a very generous > donation by Amazon. Using AWS donated credits we now have a fast > open-source CI system for pre-commit testing. This is intended primarily > for committers (and trusted contributors) that do not have access to a > company-private CI system. Contact me off-list if that's you. > > … > This huge donation from Amazon and builds on years of work on our CI and > testing scripts, accumulating in CASSANDRA-18145¹. > > The project now has an AWS account (controlled by ASF Infra), and into it > Amazon has provided a significant amount of credits for the purpose of our > pre-commit testing. > > In the C* project pre-commit CI has been a major hurdle and blocker to new > contributors, unaffiliated committers, and generally getting patches > committed. Too much of our momentum has been led by contributors and > committers employed in companies that can pay to run their own in-house CI > systems. It is not in the slightest their fault, nor their employers' > priority, but it has held the project back from its potential as well as > not being in the proper spirit of an open source project and community. > > Amazon has recognised this and stepped forward as a true champion of our > open-source community, providing both the AWS credits, as well as time in > helping setting it up, testing and documenting it. > > The work has also involved many steps and people the past five years! > since that first mention of a "Stand-alone Pipeline Run"². This work has > gone from having a standalone Jenkinsfile, testing images and scripts that > can be used anywhere, both pre-commit and post-commit, to a Helm chart to > deploy ci-cassandra.a.o Jenkins clones in any k8s cluster of your > choosing. This has also included getting our CI pipeline generating > aggregated test result artefact files that can easily be attached to jira > tickets, and archived permanently. These objectives you can see listed in > CASSANDRA-18137³. > > For all this special thanks goes out to Richa Mishra, Nishant Barola, > Aleksandr Volochnev, Brandon Hsieh, Himanshu Jindal, Steve Mayszak, Dmitry > Konstantinov, Maxim Muzafarov, Berenguer Blasi, David Capwell, Brandon > Williams, Ekaterina Dimitrova, Henrik Ingo, Shankar Ramaswamy, Josh > McKenzie, Stefan Miklosovic, Gavin McDonald and no doubt others I've > forgotten! Yeah, it feels like almost everyone, it has been a journey! > > It is not perfect: there is (always) room for improvement; and for now I > think next steps should be around further time+cost optimisations and > pipeline configurability. I'm hoping folk can jump in and help with > these. > > What is provided today is a way for everyone anywhere to immutably > provision a ci-cassandra.a.o clone on k8s. You avoid any need to install, > know or operate Jenkins (you do not even have to ever once open the Jenkins > UI – see below) and you do not need to wait more than a few hours to run > the "post-commit" pipeline with 140-200k tests. The correlation to > pipeline time is now how many nodes your k8s can scale to (ofc limitations > exist, but those fine details are other convo), and with the cluster > scaling down to one node you get a cost-effective pay-what-you-use > deployment. https://pre-ci.cassandra.apache.org runs on EKS, and another > instance for DataStax employees ( http://ci-cassandra.infra.datastax.com > ) has been in use on GKE for over a year now. We have a CI system that is > relatively ops-free, has the same turnaround time as CircleCI and costs > over an order of magnitude less. > This has been designed for downstream C* products to be able to re-use, > and indeed it is a setup that any project can easily use. > > The setup introduces repeatability in how we test. The Jenkinsfile, the > docker images and the test scripts, are the same everywhere, including > locally. Reproducing obscure failures in post-commit should be easier. > > If you would like to set up your own ci-cassandra clone, the docs for both > GKE are found in-tree under .jenkins/k8s/ ,and the docs for EKS coming soon. > > > There is a command line interface that can be used for both provisioning > as well as everyday CI use. > Some examples: > ``` > # Install the ci-cassandra helm chat in your current k8s context, creating > the ci-cassandra clone > > .build/run-ci --only-setup > > # Run your current branch through CI with the skinny profile. > # Waits til completion, and creates ci_summary*.html & > results_details*.tar.xz files to attach to jira > > .build/run-ci --profile skinny > > # If you don't have kubectl access to the cluster, just specify the > jenkins url and user > > .build/run-ci --profile skinny --url pre-ci.cassandra.apache.org --user > myuser > ``` > > > Huge huge praise and thanks to Amazon here: they really deserve it, they > have taken this over the finish line; but the opportunity deserves to > repeat all our gratitudes to all the very significant > ci-cassandra.apache.org donations that help us keep the project running: > NetApp, DataStax, Huawei and iland. > > > ¹) CASSANDRA-18145 – Run entire Cassandra Jenkins in any k8s (immutable > provisioning) > ²) https://lists.apache.org/thread/ch768l8y95npno9o9y130fjob1h675z8 > ³) CASSANDRA-18137 – Repeatable ci-cassandra.a.o (Epic) > > > > -- Dmitry Konstantinov
