[
https://issues.apache.org/jira/browse/HDDS-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16452:
------------------------------
Description:
h3. Finding
A checkpoint served by InterSCMGrpcService.download during an apply window
ships data at index i under stale durable index i-1; a follower installs it and
durably holds data ahead of its recorded index (CheckpointConsistent violated
across replicas), then re-applies the entry on catch-up. Reproduced through the
real concurrent apply/download APIs on a real store. For non-idempotent SCM
transactions (e.g. deleted-block accounting, corroborated by a separate tracked
issue) this diverges across SCM replicas with no automatic recovery; idempotent
transactions self-heal via replay.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-ha-tx-buffer
Guidance:
campaigns/ozone-9fbf9ee/targets/018-scm-ha-tx-buffer/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-ha-tx-buffer|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16432 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
Precondition: two threads run concurrently on a leader SCM, the Ratis apply
thread in SCMStateMachine.applyTransaction and the InterSCM gRPC download
thread in InterSCMGrpcService.download.
1. The apply thread opens an apply window: beginApplyingTransaction, then
process() stages transaction i's data via addToBuffer (the batch now holds data
i), but has not yet reached updateLatestTrxInfo, so the durable transaction
index is still i-1.
2. A follower requests a checkpoint. InterSCMGrpcService.download calls the
unguarded SCMHADBTransactionBuffer.flush(), which takes the write lock and
commits the current batch atomically: {data i, transactionInfo i-1}. The DB now
holds data i under the stale index i-1.
3. download serves that checkpoint DB. The follower installs it
(notifyInstallSnapshotFromLeader, reinitialize, transactionBuffer.init()) and
reloads the persisted index i-1 while its DB already contains data i
(CheckpointConsistent / DurableIndexMatchesData violated), then re-applies
entry i on catch-up.
For idempotent SCM transactions the replay self-heals, so the harm is masked.
For a non-idempotent replicated transaction (e.g. deleted-block accounting) the
re-apply diverges across SCM replicas with no automatic recovery.
Reproduced with a focused test that drives the real download() against a real
store and buffer with an apply in progress: on the unguarded flush() the store
is left holding data i under index i-1; the assertion fails without the fix and
passes with it.
h3. Root cause
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcService.java:
download() calls SCMHADBTransactionBuffer.flush(), which commits whatever is
in the current batch under the current (not yet advanced) transaction index.
flush() takes only the write lock; unlike flushIfNeeded it does not check
applyingTransactions, so it is not deferred during an apply. The apply window
is not atomic (addToBuffer releases the read lock on return,
updateLatestTrxInfo holds no lock), so a flush can land between staging data i
and advancing the index. HDDS-15065 added the applyingTransactions guard and
moved the periodic/monitor flush onto it, but the download caller was left on
the unguarded flush().
h3. Suggested fix
Route the download flush through the existing guarded flushIfNeeded(0) instead
of flush(). While a transaction apply is in progress it defers (returns without
committing), so the checkpoint is served from the last self consistent durable
state and the follower catches the missing transactions up through the Ratis
log. When no apply is in progress it flushes pending data as before. This
reuses the same guard HDDS-15065 established for the periodic path. See
[^HDDS-16452.001.patch].
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
A checkpoint served by InterSCMGrpcService.download during an apply window
ships data at index i under stale durable index i-1; a follower installs it and
durably holds data ahead of its recorded index (CheckpointConsistent violated
across replicas), then re-applies the entry on catch-up. Reproduced through the
real concurrent apply/download APIs on a real store. For non-idempotent SCM
transactions (e.g. deleted-block accounting, corroborated by a separate tracked
issue) this diverges across SCM replicas with no automatic recovery; idempotent
transactions self-heal via replay.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-ha-tx-buffer
Guidance:
campaigns/ozone-9fbf9ee/targets/018-scm-ha-tx-buffer/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-ha-tx-buffer|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16432 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> Inter SCM checkpoint served during an apply window ships data ahead of the
> durable index, diverging replicas
> ------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16452
> URL: https://issues.apache.org/jira/browse/HDDS-16452
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-16452.001.patch
>
>
> h3. Finding
> A checkpoint served by InterSCMGrpcService.download during an apply window
> ships data at index i under stale durable index i-1; a follower installs it
> and durably holds data ahead of its recorded index (CheckpointConsistent
> violated across replicas), then re-applies the entry on catch-up. Reproduced
> through the real concurrent apply/download APIs on a real store. For
> non-idempotent SCM transactions (e.g. deleted-block accounting, corroborated
> by a separate tracked issue) this diverges across SCM replicas with no
> automatic recovery; idempotent transactions self-heal via replay.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: scm-ha-tx-buffer
> Guidance:
> campaigns/ozone-9fbf9ee/targets/018-scm-ha-tx-buffer/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-ha-tx-buffer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16432 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition: two threads run concurrently on a leader SCM, the Ratis apply
> thread in SCMStateMachine.applyTransaction and the InterSCM gRPC download
> thread in InterSCMGrpcService.download.
> 1. The apply thread opens an apply window: beginApplyingTransaction, then
> process() stages transaction i's data via addToBuffer (the batch now holds
> data i), but has not yet reached updateLatestTrxInfo, so the durable
> transaction index is still i-1.
> 2. A follower requests a checkpoint. InterSCMGrpcService.download calls the
> unguarded SCMHADBTransactionBuffer.flush(), which takes the write lock and
> commits the current batch atomically: {data i, transactionInfo i-1}. The DB
> now holds data i under the stale index i-1.
> 3. download serves that checkpoint DB. The follower installs it
> (notifyInstallSnapshotFromLeader, reinitialize, transactionBuffer.init()) and
> reloads the persisted index i-1 while its DB already contains data i
> (CheckpointConsistent / DurableIndexMatchesData violated), then re-applies
> entry i on catch-up.
> For idempotent SCM transactions the replay self-heals, so the harm is masked.
> For a non-idempotent replicated transaction (e.g. deleted-block accounting)
> the re-apply diverges across SCM replicas with no automatic recovery.
> Reproduced with a focused test that drives the real download() against a real
> store and buffer with an apply in progress: on the unguarded flush() the
> store is left holding data i under index i-1; the assertion fails without the
> fix and passes with it.
> h3. Root cause
> hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/InterSCMGrpcService.java:
> download() calls SCMHADBTransactionBuffer.flush(), which commits whatever is
> in the current batch under the current (not yet advanced) transaction index.
> flush() takes only the write lock; unlike flushIfNeeded it does not check
> applyingTransactions, so it is not deferred during an apply. The apply window
> is not atomic (addToBuffer releases the read lock on return,
> updateLatestTrxInfo holds no lock), so a flush can land between staging data
> i and advancing the index. HDDS-15065 added the applyingTransactions guard
> and moved the periodic/monitor flush onto it, but the download caller was
> left on the unguarded flush().
> h3. Suggested fix
> Route the download flush through the existing guarded flushIfNeeded(0)
> instead of flush(). While a transaction apply is in progress it defers
> (returns without committing), so the checkpoint is served from the last self
> consistent durable state and the follower catches the missing transactions up
> through the Ratis log. When no apply is in progress it flushes pending data
> as before. This reuses the same guard HDDS-15065 established for the periodic
> path. See [^HDDS-16452.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
