Ray Chen created CASSANDRA-21581:
------------------------------------
Summary: RepairedDataInconsistenciesConfirmed reports empty vs
non-empty repaired digests while on-disk repaired data matches and repair
preview is in sync
Key: CASSANDRA-21581
URL: https://issues.apache.org/jira/browse/CASSANDRA-21581
Project: Apache Cassandra
Issue Type: Bug
Reporter: Ray Chen
h2. Environment
* Apache Cassandra 4.0.x (observed on 4.0.19)
* Cluster under incremental-repair evaluation
* Affected table has {{read_repair = NONE}}
h2. Symptom
We repeatedly observe {{RepairedDataInconsistenciesConfirmed}} / WARN from
{{RepairedDataVerifier$SimpleVerifier}} during normal application reads:
* Pattern is often *empty repaired digest on one replica* vs *non-empty
repaired digest on another* for the same partition
* Example shape from logs:
{code:java}
Detected mismatch between repaired datasets for table <ks>.<table> during read
of Read(... key=<partition-key-hex> ...).
RepairedDataTracker{digests={=[/nodeA:7000], <digest>=[/nodeB:7000]},
inconclusive=[]}
{code}
h2. Follow-up checks that do *not* show divergence
* Follow-up full repair / preview finds no divergence (preview reported "in
sync")
* {{{}read_repair = NONE{}}}, so this is unlikely explained by read-repair
healing between the alert and the preview
h2. On-disk investigation for a recent WARN
For one recent Confirmed event we took the partition key from the WARN and
inspected both replicas listed in {{{}RepairedDataTracker{}}}:
* {{nodetool getsstables}} on both nodes: the partition was present in
SSTables on both sides
* On each node the key appeared in *one repaired* SSTable and *one unrepaired*
SSTable
* {{{}sstablemetadata{}}}: repaired SSTables on both nodes shared the same
non-zero {{Repaired at}} timestamp; unrepaired had {{Repaired at: 0}}
* {{sstabledump -k <key>}} on the *repaired* SSTables: partition contents
matched across the two replicas (same clustering keys, same timestamps/TTL
metadata, same cell values)
Despite matching on-disk repaired content (at inspection time), the verifier
had reported empty vs non-empty repaired digests for that read.
h2. What we searched
We did not find a clear matching *open* Apache Cassandra Jira for this
empty-vs-non-empty Confirmed + later "in sync" / matching repaired dump pattern.
h2. Ask
Is this expected/known behavior of repaired-data tracking, or a bug / false
positive in digest verification for the already-repaired set?
Happy to attach more redacted logs and {{getsstables}} / {{sstablemetadata}} /
{{sstabledump}} snippets if useful.
h2. Notes
* Concern is that Confirmed mismatches may be misleading during incremental
repair evaluation (metric/noise vs real repaired-set divergence).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
