[
https://issues.apache.org/jira/browse/HDDS-16454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16454:
------------------------------
Description:
h3. Finding
The new-leader branch starts the datanode server with applied index below
committed index, so reports for committed-but-unapplied containers are dropped
as CONTAINER_NOT_FOUND, leaving SCM's in-memory replica-location accounting
incomplete; triggerable when leadership moves to a node whose apply lags
commit. Masked because the default WARN action never deletes the physical
replica, the NodeManager node-to-container map is retained, and
replication-manager consumers are gated on isLeaderReady and safe-mode exit, so
only self-correcting RM churn could surface in the report-interval window.
h3. Classification
* Verdict: MASKED
* Severity: Medium
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-follower-catchup
Guidance:
campaigns/ozone-9fbf9ee/targets/019-scm-follower-catchup/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-follower-catchup|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16432 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
This is a latent (masked) defect. The invariant
NoLeaderBranchStartBeforeApplied is violated; the drop of a committed replica
is masked by the default report handling, so a live surfacing needs the
unmasking window below.
1. A node whose state-machine apply lags its committed index wins a Ratis
election (leadership transfer / election requires an up-to-date log, not
applied state). Ratis fires notifyLeaderChanged, which calls
updateLeaderAndTerm(true, term), so SCMContext.isLeader() becomes true while
appliedIndex < committedIndex.
2. notifyLeaderChanged's leader branch calls
tryStartDNServerAndRefreshSafeMode(). Its gate short-circuits on isLeader(), so
the datanode protocol server starts immediately without waiting for the
committed backlog to be applied.
3. A datanode reports a replica for a container that is committed but not yet
applied (still absent from SCM's containerSet). ContainerReportHandler drops it
as CONTAINER_NOT_FOUND, so SCM's in-memory replica-location accounting for that
container is left incomplete until the next full container report.
Unmasking window: the default CONTAINER_NOT_FOUND action only logs a warning
(it does not delete the physical replica) and the NodeManager node-to-container
map is retained, and replication-manager consumers are gated on isLeaderReady
and safe-mode exit, so only self-correcting replication-manager churn within
the report-interval window could surface the incomplete accounting.
h3. Root cause
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
tryStartDNServerAndRefreshSafeMode gated the start on
scm.getScmContext().isLeader() || isFollowerCaughtUp(). isLeader() is true from
the role transition in notifyLeaderChanged, which precedes the asynchronous
apply of the committed backlog, so the leader branch bypassed the applied-index
check. The follower branch already defers correctly via isFollowerCaughtUp()
(appliedIndex >= the committed target captured at (re)join). The feature that
added the deferred follower start (HDDS-14989) left the leader path starting on
isLeader() alone.
h3. Suggested fix
Drop the isLeader() short-circuit so the datanode protocol server start is
gated purely on isFollowerCaughtUp(), the concrete appliedIndex >=
committed-target check. This holds for a new leader too (getLeaderCommitIndex
returns the leader's own commit index), so the start is deferred until the
committed backlog is applied and is driven by applyTransaction /
notifyTermIndexUpdated as catch-up completes. A node that is already caught up
when it becomes leader still starts immediately. A unit test asserts that with
isLeader() true but appliedIndex below the committed index the server does not
start, and that it starts once the backlog is applied. See
[^HDDS-16454.001.patch].
Note: notifyLeaderReady can fire while live commits are still ahead of the
accession backlog; addressing that residual is tracked separately and is out of
scope for this change.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
The new-leader branch starts the datanode server with applied index below
committed index, so reports for committed-but-unapplied containers are dropped
as CONTAINER_NOT_FOUND, leaving SCM's in-memory replica-location accounting
incomplete; triggerable when leadership moves to a node whose apply lags
commit. Masked because the default WARN action never deletes the physical
replica, the NodeManager node-to-container map is retained, and
replication-manager consumers are gated on isLeaderReady and safe-mode exit, so
only self-correcting RM churn could surface in the report-interval window.
h3. Classification
* Verdict: MASKED
* Severity: Medium
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-follower-catchup
Guidance:
campaigns/ozone-9fbf9ee/targets/019-scm-follower-catchup/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-follower-catchup|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16432 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> New SCM leader with apply lagging commit drops container reports as
> CONTAINER_NOT_FOUND
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-16454
> URL: https://issues.apache.org/jira/browse/HDDS-16454
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: HDDS-16454.001.patch
>
>
> h3. Finding
> The new-leader branch starts the datanode server with applied index below
> committed index, so reports for committed-but-unapplied containers are
> dropped as CONTAINER_NOT_FOUND, leaving SCM's in-memory replica-location
> accounting incomplete; triggerable when leadership moves to a node whose
> apply lags commit. Masked because the default WARN action never deletes the
> physical replica, the NodeManager node-to-container map is retained, and
> replication-manager consumers are gated on isLeaderReady and safe-mode exit,
> so only self-correcting RM churn could surface in the report-interval window.
> h3. Classification
> * Verdict: MASKED
> * Severity: Medium
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: scm-follower-catchup
> Guidance:
> campaigns/ozone-9fbf9ee/targets/019-scm-follower-catchup/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-follower-catchup|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16432 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> This is a latent (masked) defect. The invariant
> NoLeaderBranchStartBeforeApplied is violated; the drop of a committed replica
> is masked by the default report handling, so a live surfacing needs the
> unmasking window below.
> 1. A node whose state-machine apply lags its committed index wins a Ratis
> election (leadership transfer / election requires an up-to-date log, not
> applied state). Ratis fires notifyLeaderChanged, which calls
> updateLeaderAndTerm(true, term), so SCMContext.isLeader() becomes true while
> appliedIndex < committedIndex.
> 2. notifyLeaderChanged's leader branch calls
> tryStartDNServerAndRefreshSafeMode(). Its gate short-circuits on isLeader(),
> so the datanode protocol server starts immediately without waiting for the
> committed backlog to be applied.
> 3. A datanode reports a replica for a container that is committed but not yet
> applied (still absent from SCM's containerSet). ContainerReportHandler drops
> it as CONTAINER_NOT_FOUND, so SCM's in-memory replica-location accounting for
> that container is left incomplete until the next full container report.
> Unmasking window: the default CONTAINER_NOT_FOUND action only logs a warning
> (it does not delete the physical replica) and the NodeManager
> node-to-container map is retained, and replication-manager consumers are
> gated on isLeaderReady and safe-mode exit, so only self-correcting
> replication-manager churn within the report-interval window could surface the
> incomplete accounting.
> h3. Root cause
> hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java:
> tryStartDNServerAndRefreshSafeMode gated the start on
> scm.getScmContext().isLeader() || isFollowerCaughtUp(). isLeader() is true
> from the role transition in notifyLeaderChanged, which precedes the
> asynchronous apply of the committed backlog, so the leader branch bypassed
> the applied-index check. The follower branch already defers correctly via
> isFollowerCaughtUp() (appliedIndex >= the committed target captured at
> (re)join). The feature that added the deferred follower start (HDDS-14989)
> left the leader path starting on isLeader() alone.
> h3. Suggested fix
> Drop the isLeader() short-circuit so the datanode protocol server start is
> gated purely on isFollowerCaughtUp(), the concrete appliedIndex >=
> committed-target check. This holds for a new leader too (getLeaderCommitIndex
> returns the leader's own commit index), so the start is deferred until the
> committed backlog is applied and is driven by applyTransaction /
> notifyTermIndexUpdated as catch-up completes. A node that is already caught
> up when it becomes leader still starts immediately. A unit test asserts that
> with isLeader() true but appliedIndex below the committed index the server
> does not start, and that it starts once the backlog is applied. See
> [^HDDS-16454.001.patch].
> Note: notifyLeaderReady can fire while live commits are still ahead of the
> accession backlog; addressing that residual is tracked separately and is out
> of scope for this change.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
