Nivy Kani created CASSANDRA-21649:
-------------------------------------
Summary: Ensure SSTables are deleted in order of last-updated
timestamp
Key: CASSANDRA-21649
URL: https://issues.apache.org/jira/browse/CASSANDRA-21649
Project: Apache Cassandra
Issue Type: Bug
Reporter: Nivy Kani
Assignee: Nivy Kani
When a node restarts after crashing, we need to clean up any partially-deleted
SSTables, but we do a safety check on REMOVE log records to verify that the
last-modified timestamps match the actual files on disk. If the timestamps
don’t match, we throw an error that the log is corrupted, and the node doesn’t
start.
The comment in LogFile.verifyRecord() states: "Because we delete files from
oldest to newest, the latest update time should always match.” But this isn’t
always true; regular SSTable deletion after compaction always deletes DATA
files first, and doesn’t explicitly set the deletion order of other files.
So we can have this scenario:
1. SSTable files are written, and there’s some tiny delay between the
written-timestamp of each file.
2. Compaction succeeds. SSTable files can be deleted now, and we write a REMOVE
record with the timestamp of the most-recently-updated file of each SSTable.
3. SSTableTidier starts deleting the SSTable files. Problem: it can delete the
files in non-timestamp order.
4. Halfway through, the node crashes. On restart, LogFile.verifyRecord() goes
through each REMOVE record to compare timestamps. It fails because the
last-written file was deleted, so the most-recent-timestamp is different now.
5. Now the node won’t start up, and some SSTable files are leaked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
