> On Apr 11, 2025, at 1:15 PM, Jon Haddad <j...@rustyrazorblade.com> wrote:
>
>
> I also keep running up against my concern about treating object store as a
> write back cache instead of write through. "Tiering" data off has real
> consequences for the user, the big one being data loss, especially with
> regards to tombstones. I think this is a pretty serious foot gun. It's the
> same problem we originally had with JBOD, where we could have tombstones on
> one disk and the shadowed data on the other. Losing one disk results in data
> getting resurrected. Anthony covered it in a blog post [1] and I believe
> CASSANDRA-6696 was the JIRA that addressed the problem. Introducing tiering
> would essentially bring this problem back.
If you lose one disk, you shoot the instance. We have to stop pretending you
can have partial failures. That’s it. That’s the fix. You don’t get to lose
part of a machine and pretend it’s still viable. Just like losing a a commit
log segment or losing an object in a bucket, if you lose one object, you throw
it away or you’ve resurrected data / violated consistency.
