Nivy Kani created CASSANDRA-21694:
-------------------------------------
Summary: Corrupt sstables can be zero-copy streamed to replicas
Key: CASSANDRA-21694
URL: https://issues.apache.org/jira/browse/CASSANDRA-21694
Project: Apache Cassandra
Issue Type: Bug
Reporter: Nivy Kani
Assignee: Nivy Kani
Currently, the zero-copy path for streaming sstables differs from the
chunk-based streaming path in that it doesn’t verify checksums.
This means we have a potential data-loss bug where an sstable becomes corrupt
on node A, and then zero-copy streams that sstable to node B during its
bootstrap, and now the corruption exists on multiple replicas. Even though
compaction catches the corruption (and excludes that sstable from future
compactions, so the CRC itself isn’t rewritten), it doesn’t stop the sstable
from being streamed.
The fix is to check Digest.crc32 after zero-copy streaming but before the
sstable is made visible. The digest is written when the sstable is first
created on the sender, so it describes the original bytes. We maintain the
benefits of zero-copy streaming, since we still never parse the sstable
contents. It also makes sense to compute the checksum on only the receiver’s
side so that the sender doesn’t have to read the file into user space. The cost
is one re-read of the Data.db file on the receiver, which will likely be in the
cache since we just wrote it.
Note that although we always write a Digest.32, nothing currently fails if it
doesn’t exist, so this check will be fail-open. There’s also a known gap where
this won’t catch if any files other than Data.db is corrupt, or if the CRC was
written for a corrupt table to begin with.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
