Yep. So I think that we should work "backwards". First we should probably do manifest-v2.json which is way more comfortable to work with, contains hashes of files, whole schema, each file ... any other metadata.
Once this is in place then we would do "copying to a local disk" with this deduplication in place and eventually we would abstract away the local disk to work with anything else. On Thu, Oct 8, 2026 at 8:35 PM Isaac Reath <[email protected]> wrote: > > > We have solved this already in our tooling, each snapshot has a > manifest containing all SSTables it logically consists of. So if you > go to remove a snapshot, you can remove a particular SSTable only in > case it is not referenced in any other manifest. Otherwise your other > snapshot would be lacking sstables. > > This is pretty much how Medusa handles this. Medusa uploads a manifest with > each backup to S3 with the files in the backup. SSTables are only removed > from S3 when `medusa purge` runs. `medusa purge` in turn deletes the files > which are not referenced in any active backup manifest (and have had their > configurable grace period expire to ensure that purge doesn't race with an > ongoing backup). I think any tooling we build in-project for this (cassandra, > sidecar or otherwise) should consider a similar approach. > > On Thu, Oct 8, 2026 at 2:16 PM Štefan Miklošovič <[email protected]> > wrote: >> >> Regarding snapshots, I want to say there is one conceptual "problem" >> which seems to be more complicated than it seems. >> >> Currently, a snapshot is a bunch of hard links. Hardlinks don't >> translate to S3. So we need to copy. Alright, so imagine we have this >> snapshot: >> >> snap1 >> >> SSTable1 >> SSTable2 >> SSTable3 >> >> snap2 >> >> SSTable3 >> SSTable4 >> SSTable5 >> >> snap1 is uploaded, then snap2 is going to be, sstable 3 is in s3 >> already so we skip that, then upload sstable 4 and 5. >> >> Now you go to remove snap2. But you can not remove sstable 3. Because >> it is logically contained in snap1. Ha. >> >> In other words, you would need to have some global way to track this. >> >> We have solved this already in our tooling, each snapshot has a >> manifest containing all SSTables it logically consists of. So if you >> go to remove a snapshot, you can remove a particular SSTable only in >> case it is not referenced in any other manifest. Otherwise your other >> snapshot would be lacking sstables. >> >> So this "just upload snapshots to s3" is way more involved than it >> seems, we would need to solve stuff like this. >> >> Or some other approach can be taken? I have not done extensive >> research on how other tooling deals with this. Maybe there is some >> elegant workaround. >> >> On Thu, Oct 8, 2026 at 7:59 PM Štefan Miklošovič <[email protected]> >> wrote: >> > >> > These are all valid points you raised. I think that CEP-36 can be >> > "limited" to just providing a pluggable way for ChannelProxy. Serving >> > from S3 directly or not, there will still need to be a way to get a >> > cloud-specific implementation of ChannelProxy for reads / write of any >> > kind. Later on we can decide the overall architecture of that but I >> > would reserve it for another CEP, that work will most probably be way >> > more involved than "just" a ChannelProxy creation. >> > >> > If ChannelProxy is pluggable already, we can start to use this for >> > snapshot uploading _if somebody ever wants to go that route_, I would >> > just need to tweak the snapshotting stuff to be ChannelProxy-aware >> > instead of on "File"-s. The default impl would be on FileChannel as we >> > know it. >> > >> > On Thu, Oct 8, 2026 at 5:02 PM Jon Haddad <[email protected]> wrote: >> > > >> > > I didn't realize the CEP hadn't been updated. I think I was mixing up >> > > the convo on the ML and what was actually in the CEP. I had remembered >> > > it behaving much closer to clickhouse, where it keeps the data on disk >> > > and pushes the compacted files to S3. I don't think serving directly >> > > off S3 is going to be much use to anyone. The latency will be awful, >> > > and they issue a backoff warning fairly quickly. In my mind it only >> > > makes sense to treat S3 as the authority and cache SStables to local >> > > disk. I think the Astra folks may have done something along these lines, >> > > I'd be willing to bet there's a ton of knowledge there as they >> > > operationalized it successfully. >> > > >> > > Anyways - the reason I asked my question, was to frame it in the context >> > > of a longer term project strategy. Asking more directly - If we're >> > > going to be pushing and pulling from S3 during compaction, and it's a >> > > major feature, would that go in Sidecar? It certainly could, but I'm >> > > concerned about having a critical component of C* living it's own >> > > independent lifecycle. >> > > >> > > Example of why I get concerned: >> > > https://the-asf.slack.com/archives/C07V0FK0M2P/p1788964591823929 >> > > >> > > Imo, having 2 projects that are tightly coupled, but developed >> > > independently, is going to be fairly high risk. I really don't like >> > > having responsibility spread this way, and I probably wouldn't want to >> > > run it in prod. >> > > >> > > Jon >> > > >> > > >> > > >> > > On Wed, Oct 7, 2026 at 7:15 AM Štefan Miklošovič >> > > <[email protected]> wrote: >> > >> >> > >> I have a draft of CEP-36 here (1). S3 deps + proxy impl is meant to be >> > >> implemented in a different project (2) which once built is added to >> > >> Cassandra via custom ClassLoader so it is all encapsulated and we can >> > >> not have any dependency clashes. >> > >> >> > >> There is the impl of s3 proxy with e2e tests. It shows how Cassandra >> > >> starts to serve reads from s3. When it comes to writes, these SSTables >> > >> will be created locally as they are not and then "offloaded" to S3 for >> > >> reads. Once SSTable is created it is immutable so we can upload it and >> > >> then switch SSTableReader to read via S3 proxy. >> > >> >> > >> At least that is the idea. Heavy demo. Dragons everywhere. AI-driven a >> > >> lot. It more _does not work_ than does :D But it genuinely starts to >> > >> read from S3 via proxy which is quite ... promising, but I think that >> > >> if we want to seriously consider CEP-36 and implement it fully that is >> > >> a serious piece of work for two quarters :D >> > >> >> > >> Give me some time here please, like a week or two, I would like to >> > >> explore this in more detail. Maybe it will be better to modify >> > >> SnapshotManager to work with ChannelProxy instead of File and when >> > >> taking snapshots we would use S3 ChannelProxy plus we would tweak the >> > >> hardlinking bit. >> > >> >> > >> You can of course do this all in Sidecar, I just want to see where >> > >> this gets us. You would have two ways to choose from (and CEP-36 >> > >> building blocks would be in place to extend on further). >> > >> >> > >> (1) https://github.com/apache/cassandra/pull/5238/changes >> > >> (2) https://github.com/smiklosovic/cassandra-s3-channel-proxy >> > >> >> > >> On Wed, Oct 7, 2026 at 3:30 PM Isaac Reath <[email protected]> wrote: >> > >> > >> > >> > In fairness, CEP-36 does explicitly call out that anything that uses >> > >> > a hard link will no longer work once data is written in an >> > >> > alternative location: >> > >> > >> > >> > > Once triggered any data written to alternative locations will not >> > >> > > be included in the snapshotting, incremental backups or other >> > >> > > processes that use hard file links. >> > >> > >> > >> > In a world where we have CEP-36 and an S3-backed implementation of >> > >> > ChannelProxy, I would imagine that a call to the >> > >> > ColumnFamilyStore.takeSnapshot function would either no-op (sticking >> > >> > to what is defined in the CEP), wrap a call to take an on-demand >> > >> > backup via AWS Backup, or do something similar to running aws s3 >> > >> > sync. In this world, if you wanted to take a backup across a whole >> > >> > cluster, you could still use Sidecar to manage the coordination. Or >> > >> > you could just use continuous backups on your S3 buckets and call it >> > >> > a day. Depends on the semantics and use-cases of what we're looking >> > >> > for out of a backup solution. >> > >> > >> > >> > CEP-36 offers details around how to get the ChannelProxy interface in >> > >> > place, but leaves the details of how to implement an object >> > >> > store-backed version of the storage engine as an exercise for the >> > >> > reader. I think if we were to get CEP-36 voted on and implemented, >> > >> > we'd need a follow-up CEP that dives into the S3 implementation, or >> > >> > otherwise we'd need to expand the current scope of CEP-36 to cover it. >> > >> > >> > >> > On Wed, Oct 7, 2026 at 6:31 AM Štefan Miklošovič >> > >> > <[email protected]> wrote: >> > >> >> >> > >> >> BTW I am still not in favor of putting this to Cassandra, just have >> > >> >> snapshots in Sidecar with its own endpoints, it is way easier that >> > >> >> way. I am pretty tired at this point to go through these discussions >> > >> >> all over again yielding no actual resolution of what we are going to >> > >> >> do. "cloud-aware Cassandra" was so long on a roadmap I am losing hope >> > >> >> we will ever see that. The amount of work needed is enormous, we >> > >> >> would >> > >> >> need to go case by case and decide what is still to be written on a >> > >> >> local disk and what goes to the cloud. Some functionality is not even >> > >> >> there: >> > >> >> >> > >> >> 1) How do you atomically rename a file? S3 does not have the concept >> > >> >> of rename. There is some "S3 Express One Zone" which seems to support >> > >> >> RenameObject but that seems quite niche >> > >> >> 2) Hard links ... none, we would need to copy >> > >> >> 3) mmap, how is this done in S3? You can not memory map an object >> > >> >> store, no? We would need to do disk_access_mode: standard >> > >> >> 4) Latency ... >> > >> >> >> > >> >> Commit logs, hints, caches ... Accord stuff. I don't know what else, >> > >> >> all the local disk interaction which is illogical to move to cloud >> > >> >> would need to stay local. (anti)-compaction, repair ... all of that >> > >> >> stuff would be cloud-native too? The performance of that would be ... >> > >> >> tragic? >> > >> >> >> > >> >> I know it is something to aspire for and I do not want to be too >> > >> >> negative about this but I think it makes sense to stop for a while >> > >> >> and >> > >> >> actually see what this would mean and how much work it is. >> > >> >> >> > >> >> For that reason I would be doing backup / restores in Sidecar and be >> > >> >> done with that. It is a very well understood domain, s3 deps already >> > >> >> there, endpoints endpointing, a snapshot is a JMX call ... why to >> > >> >> complicate this more really. >> > >> >> >> > >> >> On Tue, Oct 6, 2026 at 9:11 PM Štefan Miklošovič >> > >> >> <[email protected]> wrote: >> > >> >> > >> > >> >> > Well ... that's quite a topic. >> > >> >> > >> > >> >> > The OP was asking specifically about backup / restore so I was >> > >> >> > replying in that spirit. >> > >> >> > >> > >> >> > One note for backup / restores, we talk about "snapshots", right? >> > >> >> > So >> > >> >> > if you back them up, then one would expect that Sidecar would also >> > >> >> > offer a way to list them, then to clear them ... It would not be >> > >> >> > "just >> > >> >> > about backup and restore". There would be a family of endpoints >> > >> >> > around >> > >> >> > snapshot management, very likely. That means we kind of duplicate >> > >> >> > what >> > >> >> > is in Cassandra already, if we do not put this into Cassandra >> > >> >> > directly, then Cassandra would not "see" the snapshots, except >> > >> >> > those >> > >> >> > taken locally. >> > >> >> > >> > >> >> > If we want to entertain CEP-36 specifically, that would change the >> > >> >> > picture quite significantly. In either case, snapshots would need >> > >> >> > to >> > >> >> > be copied over (SSTables, that is), because we can not do a >> > >> >> > hard-link >> > >> >> > from local storage to a S3 bucket. So we would need to copy that, >> > >> >> > right? If that is the case then we would need to probably throttle >> > >> >> > how >> > >> >> > fast it is uploaded to not interfere with other operations and >> > >> >> > similar. >> > >> >> > >> > >> >> > I was completely over-hauling the snapshot management in >> > >> >> > CASSANDRA-18111, together with Paulo Motta who did a great job >> > >> >> > reviewing it, everything is centralized in SnapshotManager(MBean). >> > >> >> > The >> > >> >> > only tasks touching the disk are TakeSnapshotTask, >> > >> >> > ClearSnapshotTask >> > >> >> > and SnapshotLoader which loads them. Their representation / >> > >> >> > metadata >> > >> >> > are in memory. So when you list it is not going to S3. That >> > >> >> > isolates >> > >> >> > what would need to be accommodated to talk to a cloud. So if you >> > >> >> > wanted to start to use S3 as a storage, taking CEP-36 into >> > >> >> > account, we >> > >> >> > would need to: >> > >> >> > >> > >> >> > 1) create "cloude-aware" implementation of >> > >> >> > org.apache.cassandra.io.util.File / java.nio.file.Path >> > >> >> > 2) create "S3 TakeSnapshotTaks" which would need to start to copy >> > >> >> > instead of hardlinking etc. >> > >> >> > 3) create another version of SnapshotLoader or modify it heavily as >> > >> >> > currently it is pretty-much wired to talk to local file system >> > >> >> > 4) Find some way how to "switch" between local / cloud via >> > >> >> > configuration in a complex manner >> > >> >> > >> > >> >> > I still think it holds that we would need to introduce some >> > >> >> > "class-path modularisation" like you would say, via configuration, >> > >> >> > what cloud storage provider you want to use and the related >> > >> >> > libraries >> > >> >> > would be auto-magically put into the class path. If we had this in >> > >> >> > place then CEP-36 might be potentially closer to the >> > >> >> > implementation as >> > >> >> > these building blocks would be there already. >> > >> >> > >> > >> >> > On Tue, Oct 6, 2026 at 6:32 PM Jon Haddad >> > >> >> > <[email protected]> wrote: >> > >> >> > > >> > >> >> > > For the folks who want to put this in sidecar, what are your >> > >> >> > > thoughts on how that would work with a potential future >> > >> >> > > improvement using s3 as part of its normal operations? I’m >> > >> >> > > thinking about CEP-36 specifically. >> > >> >> > > >> > >> >> > > Thanks! >> > >> >> > > Jon >> > >> >> > > >> > >> >> > > >> > >> >> > > On Tue, Oct 6, 2026 at 7:37 AM Isaac Reath >> > >> >> > > <[email protected]> wrote: >> > >> >> > >> >> > >> >> > >> > If I am not mistaken, Sidecar already exposes a set of >> > >> >> > >> > endpoints related to the restoration of data from S3 when >> > >> >> > >> > running some >> > >> >> > >> > Spark jobs, right? >> > >> >> > >> >> > >> >> > >> You are correct Stefan. >> > >> >> > >> >> > >> >> > >> I'll echo Stefan's concerns around the weight of adding a cloud >> > >> >> > >> provider dependency into Cassandra. Especially now, with the >> > >> >> > >> increasing volume of CVE discovery across all sorts of >> > >> >> > >> open-source projects, keeping the dependency scope of the core >> > >> >> > >> Cassandra process small is a distinct advantage. As an >> > >> >> > >> operator, it's much easier for me to push out patches across >> > >> >> > >> thousands of Sidecars than it is to push them across thousands >> > >> >> > >> of Cassandra nodes. >> > >> >> > >> >> > >> >> > >> I also think that from a resource isolation perspective, >> > >> >> > >> running these types of maintenance operations out-of-process >> > >> >> > >> can create much more predictable performance, especially when >> > >> >> > >> paired with OS-level resource governance tools like cgroups. >> > >> >> > >> What a dream it might be if we had out-of-process compaction or >> > >> >> > >> out-of-process Merkle tree calculation for repair. >> > >> >> > >> >> > >> >> > >> I understand your point that adding another Sidecar just for >> > >> >> > >> backups can increase operational complexity, but I'd argue that >> > >> >> > >> cassandra-sidecar is shaping up to offer far more than just >> > >> >> > >> backups. Today we have Sidecar acting as the entrypoint for >> > >> >> > >> bulk analytics, CEP-44 bringing CDC publishing to Kafka, CEP-40 >> > >> >> > >> adding zero-copy online data migration between nodes and with >> > >> >> > >> CEP-53 and CEP-62 we are getting rolling restarts, >> > >> >> > >> configuration management and a framework for executing other >> > >> >> > >> distributed jobs across a Cassandra cluster. Framed this way >> > >> >> > >> it's becoming a one-stop-shop for these types of operations. I >> > >> >> > >> also think that building on top of the operational job >> > >> >> > >> framework being added with CEP-53 could go a long way towards >> > >> >> > >> making the implementation easier. >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> On Tue, Oct 6, 2026 at 8:56 AM Victor Araujo >> > >> >> > >> <[email protected]> wrote: >> > >> >> > >>> >> > >> >> > >>> Hi Bernardo and Stefan, >> > >> >> > >>> >> > >> >> > >>> Thanks for the feedback. I agree that there are several >> > >> >> > >>> benefits to having backup and restore in Sidecar, especially >> > >> >> > >>> regarding isolation, dependencies, and being able to evolve >> > >> >> > >>> the feature independently from Cassandra itself. I would be >> > >> >> > >>> happy to explore and contribute to that direction, potentially >> > >> >> > >>> in my free time, and I may also be able to bring this >> > >> >> > >>> discussion to my company later. >> > >> >> > >>> >> > >> >> > >>> My main concern is more about the entry point for users. In my >> > >> >> > >>> current environment, we are actually trying to reduce our use >> > >> >> > >>> of Sidecars and other additional processes to reduce the >> > >> >> > >>> overall footprint. I also think there may be users who would >> > >> >> > >>> prefer having a basic built-in solution in Cassandra rather >> > >> >> > >>> than having to deploy another component just to get backup and >> > >> >> > >>> restore. >> > >> >> > >>> >> > >> >> > >>> Maybe there is a middle ground here. We could have an opt-in, >> > >> >> > >>> generic backup/restore capability in Cassandra core, similar >> > >> >> > >>> in spirit to the unified repair solution, without bringing any >> > >> >> > >>> third-party cloud dependencies into Cassandra. For example, >> > >> >> > >>> the core could provide the backup/restore abstractions and >> > >> >> > >>> lifecycle, with a pluggable storage implementation provided by >> > >> >> > >>> the user or another component. This could serve as a simple >> > >> >> > >>> entry-level solution, while a more complete implementation >> > >> >> > >>> with cloud-specific integrations could live in Sidecar. >> > >> >> > >>> >> > >> >> > >>> I don't know yet whether this split makes sense from an >> > >> >> > >>> architectural perspective, but I think it could be worth >> > >> >> > >>> exploring. I would be interested in hearing what others think >> > >> >> > >>> about this approach. >> > >> >> > >>> >> > >> >> > >>> Thanks, >> > >> >> > >>> Victor >> > >> >> > >>> >> > >> >> > >>> >> > >> >> > >>> Em ter., 6 de out. de 2026 às 07:49, Štefan Miklošovič >> > >> >> > >>> <[email protected]> escreveu: >> > >> >> > >>>> >> > >> >> > >>>> I also think that we should investigate how to integrate >> > >> >> > >>>> backup / >> > >> >> > >>>> restores in Sidecar rather than doing it in Cassandra. >> > >> >> > >>>> Dependency-wise, I think Sidecar already depends on s3 so >> > >> >> > >>>> that is for >> > >> >> > >>>> free already. If I am not mistaken, Sidecar already exposes a >> > >> >> > >>>> set of >> > >> >> > >>>> endpoints related to the restoration of data from S3 when >> > >> >> > >>>> running some >> > >> >> > >>>> Spark jobs, right? So this would be just a natural extension >> > >> >> > >>>> of that - >> > >> >> > >>>> a set of endpoints backuping to and restoring from an S3 >> > >> >> > >>>> bucket. >> > >> >> > >>>> >> > >> >> > >>>> However, I think that we should make it generic enough in >> > >> >> > >>>> such a way >> > >> >> > >>>> that if somebody wants to do this against Azure or GCP they >> > >> >> > >>>> could. We >> > >> >> > >>>> should not "wire" S3 only. I am not sure how to do this >> > >> >> > >>>> though. These >> > >> >> > >>>> deps are quite "heavy", depending on their own dependency >> > >> >> > >>>> chain ... I >> > >> >> > >>>> do not have good experience with mixing various cloud >> > >> >> > >>>> libraries. >> > >> >> > >>>> >> > >> >> > >>>> What we do in our backup / restore solution is that we >> > >> >> > >>>> modularize it >> > >> >> > >>>> in such a way that a user can choose what cloud they go to >> > >> >> > >>>> deal with >> > >> >> > >>>> and they have only that cloud-specific libraries on the class >> > >> >> > >>>> path. >> > >> >> > >>>> >> > >> >> > >>>> On Tue, Oct 6, 2026 at 12:14 PM Bernardo Botella >> > >> >> > >>>> <[email protected]> wrote: >> > >> >> > >>>> > >> > >> >> > >>>> > Hi Victor! >> > >> >> > >>>> > >> > >> >> > >>>> > Thanks for bringing up this discussion 😊 >> > >> >> > >>>> > >> > >> >> > >>>> > I completely agree that a first citizen backup and restore >> > >> >> > >>>> > mechanism in Cassandra would be great. >> > >> >> > >>>> > >> > >> >> > >>>> > I’d like to explore a little bit more the solution >> > >> >> > >>>> > involving Sidecar. While it is true that it would “force” >> > >> >> > >>>> > you to have yet another component running, it definitely >> > >> >> > >>>> > comes with a lot of benefits besides the current other >> > >> >> > >>>> > features it provides (control plane, analytics library, CDC >> > >> >> > >>>> > over Kafka, to name some). >> > >> >> > >>>> > >> > >> >> > >>>> > It allows for a more modular approach to a Cassandra >> > >> >> > >>>> > deployment. In my humble opinion, having smaller, more >> > >> >> > >>>> > specialized processes is the direction we should be >> > >> >> > >>>> > following, as opposed to keep making the core of the >> > >> >> > >>>> > project bigger and more complex. >> > >> >> > >>>> > It allows for a faster iteration in terms of software. When >> > >> >> > >>>> > it comes to version upgrades, it is definitely way easier >> > >> >> > >>>> > to update just the sidecar than the entire Cassandra core. >> > >> >> > >>>> > It is also less risky. >> > >> >> > >>>> > Blast radius is also smaller. It is better to have the >> > >> >> > >>>> > Sidecar affected if a backup process >> > >> >> > >>>> > stalls/breaks/misbehaves than the whole Cassandra core. >> > >> >> > >>>> > >> > >> >> > >>>> > >> > >> >> > >>>> > That’s my two cents! >> > >> >> > >>>> > Bernardo >> > >> >> > >>>> > >> > >> >> > >>>> > From: Victor Araujo <[email protected]> >> > >> >> > >>>> > Date: Monday, 5 October 2026 at 13:34 >> > >> >> > >>>> > To: [email protected] <[email protected]> >> > >> >> > >>>> > Subject: [DISCUSS] Backup and restore in Cassandra >> > >> >> > >>>> > >> > >> >> > >>>> > Hi everyone, >> > >> >> > >>>> > >> > >> >> > >>>> > I would like to discuss the possibility of having a >> > >> >> > >>>> > built-in backup and restore mechanism in Cassandra, >> > >> >> > >>>> > including incremental backups and support for external >> > >> >> > >>>> > object storage such as S3. The main goal would be to >> > >> >> > >>>> > provide a supported solution while keeping the >> > >> >> > >>>> > implementation provider-agnostic and avoiding unnecessary >> > >> >> > >>>> > operational overhead. >> > >> >> > >>>> > >> > >> >> > >>>> > I initially discussed this on Slack and was pointed to >> > >> >> > >>>> > CEP-36 and the previous discussions around external storage >> > >> >> > >>>> > and snapshots. We also discussed whether S3 would be too >> > >> >> > >>>> > provider-specific, and whether Cassandra Sidecar could be >> > >> >> > >>>> > part of the solution. My concern with Sidecar is that >> > >> >> > >>>> > introducing another component would increase the >> > >> >> > >>>> > operational footprint, which goes against one of the goals >> > >> >> > >>>> > of this work. I also have a small PoC on top of trunk >> > >> >> > >>>> > exploring some of these ideas. >> > >> >> > >>>> > >> > >> >> > >>>> > I'd be interested in hearing other opinions and any >> > >> >> > >>>> > previous work I should look at. If there is interest in >> > >> >> > >>>> > pursuing this, would a draft CEP be a useful next step so >> > >> >> > >>>> > we can start working on the design in parallel? >> > >> >> > >>>> > >> > >> >> > >>>> > Link of the first slack discussion about this topic: >> > >> >> > >>>> > https://the-asf.slack.com/archives/CK23JSY2K/p1789021792614639 >> > >> >> > >>>> > >> > >> >> > >>>> > Thanks, >> > >> >> > >>>> > Victor Cavichioli
