[
https://issues.apache.org/jira/browse/CASSANDRA-21018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18042729#comment-18042729
]
David commented on CASSANDRA-21018:
-----------------------------------
While investigating the problem, I noticed some areas in the code that appears
related.
*Root Cause Analysis:*
A DELETE ... AND rank >= ? produces a half-open range tombstone with a null
start or end bound. {{DeletionInfo.clone}} and {{RangeTombstoneList.clone}}
attempts to clone both bounds unconditionally. {{ByteBufferCloner.clone}}'s
{{assert value != null}} fails on the null bound. The stack trace in the report
confirms these methods were executed and where the assertion was thrown. No
other functions on the execution path guard or alter these null bounds; the
root cause is entirely in {{RangeTombstoneList.clone}} invoking
{{ByteBufferCloner.clone}} without null checks.
*Details:*
# In file {{src/java/org/apache/cassandra/db/RangeTombstoneList.java}},
function {{clone}} ([lines
118-122|https://github.com/apache/cassandra/blob/d48bfcce442489de45797ecdf3bde7574ade42d5/src/java/org/apache/cassandra/db/RangeTombstoneList.java#L120]):
It unconditionally calls {{cloner.clone(starts[i])}} and
{{cloner.clone(ends[i])}} even when one of those may be null for half-open
range tombstones. This leads to passing null into the cloner.
# In file {{src/java/org/apache/cassandra/utils/memory/ByteBufferCloner.java}},
function {{clone}}([line
73|https://github.com/apache/cassandra/blob/d48bfcce442489de45797ecdf3bde7574ade42d5/src/java/org/apache/cassandra/utils/memory/ByteBufferCloner.java#L73]):
It asserts {{value != null}}, so providing a null buffer triggers an
AssertionError.
*Potential remediation:*
Handle half-open tombstones: allow null bounds:
{code:java}
ByteBuffer start = (starts[i] == null) ? null : cloner.clone(starts[i]);
ByteBuffer end = (ends[i] == null) ? null : cloner.clone(ends[i]);
{code}
Let me know if this matches what you're seeing on your end.
> AssertionError in ByteBufferCloner.clone due to Range Tombstone on clustering
> key DELETE (4.1.10)
> -------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21018
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21018
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
> Reporter: Tobias Bohn
> Priority: Normal
> Fix For: 4.1.x, 5.0.x, 5.x
>
>
> h2. *Summary*
> In Apache Cassandra *4.1.10* an assertion error occurs reproducibly when
> executing a {{DELETE}} statement that produces a {*}range tombstone on the
> clustering key{*}.
> The crash happens inside the write path while applying the mutation on
> replicas, specifically in {{ByteBufferCloner.clone()}} invoked from
> {{{}RangeTombstoneList.clone(){}}}.
> The problem persists:
> * after {{nodetool scrub}} on all affected tables,
> * after deleting all hints,
> * even after completely *dropping and recreating* the keyspaces and tables.
> It only appears when executing a {{{}DELETE WHERE <partition key> AND
> <clustering key> >= ?{}}}.
> Removing this DELETE from the workload makes the issue disappear completely,
> even under heavy concurrent writes.
> This suggests a *bug in the handling of range tombstones* in the write path
> of Cassandra 4.1.x.
> h2. *Environment*
> * Cassandra version: *4.1.10*
> * 3-node cluster, RF = 3
> * No hints pending
> * Reproducible with clean keyspace & empty datasets
> * Client performs parallel INSERTs and one DELETE with a clustering key range
> h2. *Table Schema*
> {code:java}
> CREATE TABLE kkav3.ranked_kka_products (
> left text,
> rank int,
> kka_factor double,
> kka_factor_boosted double,
> left_factor double,
> left_factor_boosted double,
> left_picks int,
> right text,
> right_factor double,
> right_factor_boosted double,
> right_picks int,
> shared_picks int,
> shared_picks_boosted int,
> calculation_period int,
> last_update bigint,
> PRIMARY KEY ((left), rank)
> ) WITH CLUSTERING ORDER BY (rank ASC); {code}
> h2. *Queries that trigger the bug*
> Works fine alone:
> {code:java}
> INSERT INTO kkav3.ranked_kka_products (...) VALUES (...);{code}
> Triggers the bug (range tombstone on clustering key):
> {code:java}
> DELETE FROM kkav3.ranked_kka_products WHERE left = ? AND rank >= ?;{code}
> As soon as multiple clients insert rows and execute the above DELETE
> concurrently, Cassandra nodes throw an AssertionError inside the mutation
> processing path.
> h2. *Stacktrace*
> {code:java}
> ERROR [MutationStage-6] 2025-11-13 11:48:24,335 JVMStabilityInspector.java:68
> - Exception in thread Thread[MutationStage-6,10,SharedPool]
> java.lang.RuntimeException: java.lang.AssertionError
> at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:108)
> at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
> at
> org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:430)
> at
> org.apache.cassandra.concurrent.ExecutionFailure$1.run(ExecutionFailure.java:133)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.AssertionError: null
> at
> org.apache.cassandra.utils.memory.ByteBufferCloner.clone(ByteBufferCloner.java:73)
> at
> org.apache.cassandra.db.RangeTombstoneList.clone(RangeTombstoneList.java:131)
> at
> org.apache.cassandra.db.RangeTombstoneList.clone(RangeTombstoneList.java:120)
> at
> org.apache.cassandra.db.MutableDeletionInfo.clone(MutableDeletionInfo.java:91)
> at
> org.apache.cassandra.db.MutableDeletionInfo.clone(MutableDeletionInfo.java:33)
> at
> org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDeltaInternal(AtomicBTreePartition.java:132)
> at
> org.apache.cassandra.db.partitions.AtomicBTreePartition.addAllWithSizeDelta(AtomicBTreePartition.java:192)
> at
> org.apache.cassandra.db.memtable.SkipListMemtable.put(SkipListMemtable.java:135)
> at
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1424)
> at
> org.apache.cassandra.db.CassandraTableWriteHandler.write(CassandraTableWriteHandler.java:40)
> at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:672)
> at org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:489)
> at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:228)
> at org.apache.cassandra.hints.Hint.applyFuture(Hint.java:109)
> at
> org.apache.cassandra.hints.HintVerbHandler.doVerb(HintVerbHandler.java:116)
> at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
> at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
> ... 6 common frames omitted{code}
>
> h2. *Observed Behavior*
> * Node logs show *AssertionError* during mutation apply.
> * Tracing reports failing mutations with {{{}FAILURE_RSP{}}}.
> * Only operations involving the range delete ({{{}rank >= ?{}}}) cause the
> assertion.
> * Removing the DELETE eliminates the issue entirely.
> h2. *Expected Behavior*
> * Range deletes on clustering key ({{{}DELETE ... AND rank >= ?{}}}) should
> be processed safely.
> * No assertion failures during normal write path operations.
> * Tombstone handling in {{RangeTombstoneList.clone()}} should not produce
> null buffer conditions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
