I have a draft of CEP-36 here (1). S3 deps + proxy impl is meant to be implemented in a different project (2) which once built is added to Cassandra via custom ClassLoader so it is all encapsulated and we can not have any dependency clashes.
There is the impl of s3 proxy with e2e tests. It shows how Cassandra starts to serve reads from s3. When it comes to writes, these SSTables will be created locally as they are not and then "offloaded" to S3 for reads. Once SSTable is created it is immutable so we can upload it and then switch SSTableReader to read via S3 proxy. At least that is the idea. Heavy demo. Dragons everywhere. AI-driven a lot. It more _does not work_ than does :D But it genuinely starts to read from S3 via proxy which is quite ... promising, but I think that if we want to seriously consider CEP-36 and implement it fully that is a serious piece of work for two quarters :D Give me some time here please, like a week or two, I would like to explore this in more detail. Maybe it will be better to modify SnapshotManager to work with ChannelProxy instead of File and when taking snapshots we would use S3 ChannelProxy plus we would tweak the hardlinking bit. You can of course do this all in Sidecar, I just want to see where this gets us. You would have two ways to choose from (and CEP-36 building blocks would be in place to extend on further). (1) https://github.com/apache/cassandra/pull/5238/changes (2) https://github.com/smiklosovic/cassandra-s3-channel-proxy On Wed, Oct 7, 2026 at 3:30 PM Isaac Reath <[email protected]> wrote: > > In fairness, CEP-36 does explicitly call out that anything that uses a hard > link will no longer work once data is written in an alternative location: > > > Once triggered any data written to alternative locations will not be > > included in the snapshotting, incremental backups or other processes that > > use hard file links. > > In a world where we have CEP-36 and an S3-backed implementation of > ChannelProxy, I would imagine that a call to the > ColumnFamilyStore.takeSnapshot function would either no-op (sticking to what > is defined in the CEP), wrap a call to take an on-demand backup via AWS > Backup, or do something similar to running aws s3 sync. In this world, if you > wanted to take a backup across a whole cluster, you could still use Sidecar > to manage the coordination. Or you could just use continuous backups on your > S3 buckets and call it a day. Depends on the semantics and use-cases of what > we're looking for out of a backup solution. > > CEP-36 offers details around how to get the ChannelProxy interface in place, > but leaves the details of how to implement an object store-backed version of > the storage engine as an exercise for the reader. I think if we were to get > CEP-36 voted on and implemented, we'd need a follow-up CEP that dives into > the S3 implementation, or otherwise we'd need to expand the current scope of > CEP-36 to cover it. > > On Wed, Oct 7, 2026 at 6:31 AM Štefan Miklošovič <[email protected]> > wrote: >> >> BTW I am still not in favor of putting this to Cassandra, just have >> snapshots in Sidecar with its own endpoints, it is way easier that >> way. I am pretty tired at this point to go through these discussions >> all over again yielding no actual resolution of what we are going to >> do. "cloud-aware Cassandra" was so long on a roadmap I am losing hope >> we will ever see that. The amount of work needed is enormous, we would >> need to go case by case and decide what is still to be written on a >> local disk and what goes to the cloud. Some functionality is not even >> there: >> >> 1) How do you atomically rename a file? S3 does not have the concept >> of rename. There is some "S3 Express One Zone" which seems to support >> RenameObject but that seems quite niche >> 2) Hard links ... none, we would need to copy >> 3) mmap, how is this done in S3? You can not memory map an object >> store, no? We would need to do disk_access_mode: standard >> 4) Latency ... >> >> Commit logs, hints, caches ... Accord stuff. I don't know what else, >> all the local disk interaction which is illogical to move to cloud >> would need to stay local. (anti)-compaction, repair ... all of that >> stuff would be cloud-native too? The performance of that would be ... >> tragic? >> >> I know it is something to aspire for and I do not want to be too >> negative about this but I think it makes sense to stop for a while and >> actually see what this would mean and how much work it is. >> >> For that reason I would be doing backup / restores in Sidecar and be >> done with that. It is a very well understood domain, s3 deps already >> there, endpoints endpointing, a snapshot is a JMX call ... why to >> complicate this more really. >> >> On Tue, Oct 6, 2026 at 9:11 PM Štefan Miklošovič <[email protected]> >> wrote: >> > >> > Well ... that's quite a topic. >> > >> > The OP was asking specifically about backup / restore so I was >> > replying in that spirit. >> > >> > One note for backup / restores, we talk about "snapshots", right? So >> > if you back them up, then one would expect that Sidecar would also >> > offer a way to list them, then to clear them ... It would not be "just >> > about backup and restore". There would be a family of endpoints around >> > snapshot management, very likely. That means we kind of duplicate what >> > is in Cassandra already, if we do not put this into Cassandra >> > directly, then Cassandra would not "see" the snapshots, except those >> > taken locally. >> > >> > If we want to entertain CEP-36 specifically, that would change the >> > picture quite significantly. In either case, snapshots would need to >> > be copied over (SSTables, that is), because we can not do a hard-link >> > from local storage to a S3 bucket. So we would need to copy that, >> > right? If that is the case then we would need to probably throttle how >> > fast it is uploaded to not interfere with other operations and >> > similar. >> > >> > I was completely over-hauling the snapshot management in >> > CASSANDRA-18111, together with Paulo Motta who did a great job >> > reviewing it, everything is centralized in SnapshotManager(MBean). The >> > only tasks touching the disk are TakeSnapshotTask, ClearSnapshotTask >> > and SnapshotLoader which loads them. Their representation / metadata >> > are in memory. So when you list it is not going to S3. That isolates >> > what would need to be accommodated to talk to a cloud. So if you >> > wanted to start to use S3 as a storage, taking CEP-36 into account, we >> > would need to: >> > >> > 1) create "cloude-aware" implementation of >> > org.apache.cassandra.io.util.File / java.nio.file.Path >> > 2) create "S3 TakeSnapshotTaks" which would need to start to copy >> > instead of hardlinking etc. >> > 3) create another version of SnapshotLoader or modify it heavily as >> > currently it is pretty-much wired to talk to local file system >> > 4) Find some way how to "switch" between local / cloud via >> > configuration in a complex manner >> > >> > I still think it holds that we would need to introduce some >> > "class-path modularisation" like you would say, via configuration, >> > what cloud storage provider you want to use and the related libraries >> > would be auto-magically put into the class path. If we had this in >> > place then CEP-36 might be potentially closer to the implementation as >> > these building blocks would be there already. >> > >> > On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad <[email protected]> wrote: >> > > >> > > For the folks who want to put this in sidecar, what are your thoughts on >> > > how that would work with a potential future improvement using s3 as part >> > > of its normal operations? I’m thinking about CEP-36 specifically. >> > > >> > > Thanks! >> > > Jon >> > > >> > > >> > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath <[email protected]> wrote: >> > >> >> > >> > If I am not mistaken, Sidecar already exposes a set of >> > >> > endpoints related to the restoration of data from S3 when running some >> > >> > Spark jobs, right? >> > >> >> > >> You are correct Stefan. >> > >> >> > >> I'll echo Stefan's concerns around the weight of adding a cloud >> > >> provider dependency into Cassandra. Especially now, with the increasing >> > >> volume of CVE discovery across all sorts of open-source projects, >> > >> keeping the dependency scope of the core Cassandra process small is a >> > >> distinct advantage. As an operator, it's much easier for me to push out >> > >> patches across thousands of Sidecars than it is to push them across >> > >> thousands of Cassandra nodes. >> > >> >> > >> I also think that from a resource isolation perspective, running these >> > >> types of maintenance operations out-of-process can create much more >> > >> predictable performance, especially when paired with OS-level resource >> > >> governance tools like cgroups. What a dream it might be if we had >> > >> out-of-process compaction or out-of-process Merkle tree calculation for >> > >> repair. >> > >> >> > >> I understand your point that adding another Sidecar just for backups >> > >> can increase operational complexity, but I'd argue that >> > >> cassandra-sidecar is shaping up to offer far more than just backups. >> > >> Today we have Sidecar acting as the entrypoint for bulk analytics, >> > >> CEP-44 bringing CDC publishing to Kafka, CEP-40 adding zero-copy online >> > >> data migration between nodes and with CEP-53 and CEP-62 we are getting >> > >> rolling restarts, configuration management and a framework for >> > >> executing other distributed jobs across a Cassandra cluster. Framed >> > >> this way it's becoming a one-stop-shop for these types of operations. I >> > >> also think that building on top of the operational job framework being >> > >> added with CEP-53 could go a long way towards making the implementation >> > >> easier. >> > >> >> > >> >> > >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo <[email protected]> >> > >> wrote: >> > >>> >> > >>> Hi Bernardo and Stefan, >> > >>> >> > >>> Thanks for the feedback. I agree that there are several benefits to >> > >>> having backup and restore in Sidecar, especially regarding isolation, >> > >>> dependencies, and being able to evolve the feature independently from >> > >>> Cassandra itself. I would be happy to explore and contribute to that >> > >>> direction, potentially in my free time, and I may also be able to >> > >>> bring this discussion to my company later. >> > >>> >> > >>> My main concern is more about the entry point for users. In my current >> > >>> environment, we are actually trying to reduce our use of Sidecars and >> > >>> other additional processes to reduce the overall footprint. I also >> > >>> think there may be users who would prefer having a basic built-in >> > >>> solution in Cassandra rather than having to deploy another component >> > >>> just to get backup and restore. >> > >>> >> > >>> Maybe there is a middle ground here. We could have an opt-in, generic >> > >>> backup/restore capability in Cassandra core, similar in spirit to the >> > >>> unified repair solution, without bringing any third-party cloud >> > >>> dependencies into Cassandra. For example, the core could provide the >> > >>> backup/restore abstractions and lifecycle, with a pluggable storage >> > >>> implementation provided by the user or another component. This could >> > >>> serve as a simple entry-level solution, while a more complete >> > >>> implementation with cloud-specific integrations could live in Sidecar. >> > >>> >> > >>> I don't know yet whether this split makes sense from an architectural >> > >>> perspective, but I think it could be worth exploring. I would be >> > >>> interested in hearing what others think about this approach. >> > >>> >> > >>> Thanks, >> > >>> Victor >> > >>> >> > >>> >> > >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič >> > >>> <[email protected]> escreveu: >> > >>>> >> > >>>> I also think that we should investigate how to integrate backup / >> > >>>> restores in Sidecar rather than doing it in Cassandra. >> > >>>> Dependency-wise, I think Sidecar already depends on s3 so that is for >> > >>>> free already. If I am not mistaken, Sidecar already exposes a set of >> > >>>> endpoints related to the restoration of data from S3 when running some >> > >>>> Spark jobs, right? So this would be just a natural extension of that - >> > >>>> a set of endpoints backuping to and restoring from an S3 bucket. >> > >>>> >> > >>>> However, I think that we should make it generic enough in such a way >> > >>>> that if somebody wants to do this against Azure or GCP they could. We >> > >>>> should not "wire" S3 only. I am not sure how to do this though. These >> > >>>> deps are quite "heavy", depending on their own dependency chain ... I >> > >>>> do not have good experience with mixing various cloud libraries. >> > >>>> >> > >>>> What we do in our backup / restore solution is that we modularize it >> > >>>> in such a way that a user can choose what cloud they go to deal with >> > >>>> and they have only that cloud-specific libraries on the class path. >> > >>>> >> > >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella >> > >>>> <[email protected]> wrote: >> > >>>> > >> > >>>> > Hi Victor! >> > >>>> > >> > >>>> > Thanks for bringing up this discussion 😊 >> > >>>> > >> > >>>> > I completely agree that a first citizen backup and restore >> > >>>> > mechanism in Cassandra would be great. >> > >>>> > >> > >>>> > I’d like to explore a little bit more the solution involving >> > >>>> > Sidecar. While it is true that it would “force” you to have yet >> > >>>> > another component running, it definitely comes with a lot of >> > >>>> > benefits besides the current other features it provides (control >> > >>>> > plane, analytics library, CDC over Kafka, to name some). >> > >>>> > >> > >>>> > It allows for a more modular approach to a Cassandra deployment. In >> > >>>> > my humble opinion, having smaller, more specialized processes is >> > >>>> > the direction we should be following, as opposed to keep making the >> > >>>> > core of the project bigger and more complex. >> > >>>> > It allows for a faster iteration in terms of software. When it >> > >>>> > comes to version upgrades, it is definitely way easier to update >> > >>>> > just the sidecar than the entire Cassandra core. It is also less >> > >>>> > risky. >> > >>>> > Blast radius is also smaller. It is better to have the Sidecar >> > >>>> > affected if a backup process stalls/breaks/misbehaves than the >> > >>>> > whole Cassandra core. >> > >>>> > >> > >>>> > >> > >>>> > That’s my two cents! >> > >>>> > Bernardo >> > >>>> > >> > >>>> > From: Victor Araujo <[email protected]> >> > >>>> > Date: Monday, 5 October 2026 at 13:34 >> > >>>> > To: [email protected] <[email protected]> >> > >>>> > Subject: [DISCUSS] Backup and restore in Cassandra >> > >>>> > >> > >>>> > Hi everyone, >> > >>>> > >> > >>>> > I would like to discuss the possibility of having a built-in backup >> > >>>> > and restore mechanism in Cassandra, including incremental backups >> > >>>> > and support for external object storage such as S3. The main goal >> > >>>> > would be to provide a supported solution while keeping the >> > >>>> > implementation provider-agnostic and avoiding unnecessary >> > >>>> > operational overhead. >> > >>>> > >> > >>>> > I initially discussed this on Slack and was pointed to CEP-36 and >> > >>>> > the previous discussions around external storage and snapshots. We >> > >>>> > also discussed whether S3 would be too provider-specific, and >> > >>>> > whether Cassandra Sidecar could be part of the solution. My concern >> > >>>> > with Sidecar is that introducing another component would increase >> > >>>> > the operational footprint, which goes against one of the goals of >> > >>>> > this work. I also have a small PoC on top of trunk exploring some >> > >>>> > of these ideas. >> > >>>> > >> > >>>> > I'd be interested in hearing other opinions and any previous work I >> > >>>> > should look at. If there is interest in pursuing this, would a >> > >>>> > draft CEP be a useful next step so we can start working on the >> > >>>> > design in parallel? >> > >>>> > >> > >>>> > Link of the first slack discussion about this topic: >> > >>>> > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 >> > >>>> > >> > >>>> > Thanks, >> > >>>> > Victor Cavichioli
