Awesome work and so glad to see this!
> On Aug 28, 2025, at 8:15 AM, Jindal, Himanshu <himan...@amazon.com> wrote:
>
> A big thank you to Mick for leading the effort. This has already made a huge
> difference for contributors, myself included, and is a great step forward for
> the community. Grateful for the collaboration!
>
> Thanks
> Himanshu
>
> From: Dmitry Konstantinov <netud...@gmail.com <mailto:netud...@gmail.com>>
> Reply-To: "dev@cassandra.apache.org <mailto:dev@cassandra.apache.org>"
> <dev@cassandra.apache.org <mailto:dev@cassandra.apache.org>>
> Date: Thursday, August 28, 2025 at 6:05 AM
> To: "dev@cassandra.apache.org <mailto:dev@cassandra.apache.org>"
> <dev@cassandra.apache.org <mailto:dev@cassandra.apache.org>>
> Subject: RE: [EXTERNAL] [ANNOUNCE] Pre-commit CI @
> pre-ci.cassandra.apache.org <http://pre-ci.cassandra.apache.org/> , an AWS
> donation
>
> CAUTION: This email originated from outside of the organization. Do not click
> links or open attachments unless you can confirm the sender and know the
> content is safe.
>
> It’s a huge milestone! Many thanks to Mick and everyone else involved in
> setting up and configuring the CI. Thanks also to Amazon and the other
> companies that donated hardware resources to keep the infrastructure running.
> This is especially helpful for committers who don’t have the resources to run
> their own CI system, like me, and it makes the contribution process more
> efficient and pleasant. Hopefully, this will lead to more contributions and
> contributors.
>
> On Wed, 27 Aug 2025 at 22:50, Mick <m...@apache.org <mailto:m...@apache.org>>
> wrote:
>>
>> tl;dr
>> I am super excited to share with you all the availability of
>> https://pre-ci.cassandra.apache.org <https://pre-ci.cassandra.apache.org/> ,
>> made possible from a very generous donation by Amazon. Using AWS donated
>> credits we now have a fast open-source CI system for pre-commit testing.
>> This is intended primarily for committers (and trusted contributors) that do
>> not have access to a company-private CI system. Contact me off-list if
>> that's you.
>>
>> …
>> This huge donation from Amazon and builds on years of work on our CI and
>> testing scripts, accumulating in CASSANDRA-18145¹.
>>
>> The project now has an AWS account (controlled by ASF Infra), and into it
>> Amazon has provided a significant amount of credits for the purpose of our
>> pre-commit testing.
>>
>> In the C* project pre-commit CI has been a major hurdle and blocker to new
>> contributors, unaffiliated committers, and generally getting patches
>> committed. Too much of our momentum has been led by contributors and
>> committers employed in companies that can pay to run their own in-house CI
>> systems. It is not in the slightest their fault, nor their employers'
>> priority, but it has held the project back from its potential as well as not
>> being in the proper spirit of an open source project and community.
>>
>> Amazon has recognised this and stepped forward as a true champion of our
>> open-source community, providing both the AWS credits, as well as time in
>> helping setting it up, testing and documenting it.
>>
>> The work has also involved many steps and people the past five years! since
>> that first mention of a "Stand-alone Pipeline Run"². This work has gone
>> from having a standalone Jenkinsfile, testing images and scripts that can be
>> used anywhere, both pre-commit and post-commit, to a Helm chart to deploy
>> ci-cassandra.a.o Jenkins clones in any k8s cluster of your choosing. This
>> has also included getting our CI pipeline generating aggregated test result
>> artefact files that can easily be attached to jira tickets, and archived
>> permanently. These objectives you can see listed in CASSANDRA-18137³.
>>
>> For all this special thanks goes out to Richa Mishra, Nishant Barola,
>> Aleksandr Volochnev, Brandon Hsieh, Himanshu Jindal, Steve Mayszak, Dmitry
>> Konstantinov, Maxim Muzafarov, Berenguer Blasi, David Capwell, Brandon
>> Williams, Ekaterina Dimitrova, Henrik Ingo, Shankar Ramaswamy, Josh
>> McKenzie, Stefan Miklosovic, Gavin McDonald and no doubt others I've
>> forgotten! Yeah, it feels like almost everyone, it has been a journey!
>>
>> It is not perfect: there is (always) room for improvement; and for now I
>> think next steps should be around further time+cost optimisations and
>> pipeline configurability. I'm hoping folk can jump in and help with these.
>>
>> What is provided today is a way for everyone anywhere to immutably provision
>> a ci-cassandra.a.o clone on k8s. You avoid any need to install, know or
>> operate Jenkins (you do not even have to ever once open the Jenkins UI – see
>> below) and you do not need to wait more than a few hours to run the
>> "post-commit" pipeline with 140-200k tests. The correlation to pipeline
>> time is now how many nodes your k8s can scale to (ofc limitations exist, but
>> those fine details are other convo), and with the cluster scaling down to
>> one node you get a cost-effective pay-what-you-use deployment.
>> https://pre-ci.cassandra.apache.org <https://pre-ci.cassandra.apache.org/>
>> runs on EKS, and another instance for DataStax employees (
>> http://ci-cassandra.infra.datastax.com
>> <http://ci-cassandra.infra.datastax.com/> ) has been in use on GKE for over
>> a year now. We have a CI system that is relatively ops-free, has the same
>> turnaround time as CircleCI and costs over an order of magnitude less.
>> This has been designed for downstream C* products to be able to re-use, and
>> indeed it is a setup that any project can easily use.
>>
>> The setup introduces repeatability in how we test. The Jenkinsfile, the
>> docker images and the test scripts, are the same everywhere, including
>> locally. Reproducing obscure failures in post-commit should be easier.
>>
>> If you would like to set up your own ci-cassandra clone, the docs for both
>> GKE are found in-tree under .jenkins/k8s/ ,and the docs for EKS coming soon.
>>
>>
>> There is a command line interface that can be used for both provisioning as
>> well as everyday CI use.
>> Some examples:
>> ```
>> # Install the ci-cassandra helm chat in your current k8s context, creating
>> the ci-cassandra clone
>>
>> .build/run-ci --only-setup
>>
>> # Run your current branch through CI with the skinny profile.
>> # Waits til completion, and creates ci_summary*.html &
>> results_details*.tar.xz files to attach to jira
>>
>> .build/run-ci --profile skinny
>>
>> # If you don't have kubectl access to the cluster, just specify the jenkins
>> url and user
>>
>> .build/run-ci --profile skinny --url pre-ci.cassandra.apache.org
>> <http://pre-ci.cassandra.apache.org/> --user myuser
>> ```
>>
>>
>> Huge huge praise and thanks to Amazon here: they really deserve it, they
>> have taken this over the finish line; but the opportunity deserves to repeat
>> all our gratitudes to all the very significant ci-cassandra.apache.org
>> <http://ci-cassandra.apache.org/> donations that help us keep the project
>> running: NetApp, DataStax, Huawei and iland.
>>
>>
>> ¹) CASSANDRA-18145 – Run entire Cassandra Jenkins in any k8s (immutable
>> provisioning)
>> ²) https://lists.apache.org/thread/ch768l8y95npno9o9y130fjob1h675z8
>> ³) CASSANDRA-18137 – Repeatable ci-cassandra.a.o (Epic)
>>
>>
>>
>
>
>
> --
> Dmitry Konstantinov
