[
https://issues.apache.org/jira/browse/HDDS-16449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16449:
------------------------------
Description:
h3. Finding
On SCM leadership transfer the new leader clears pending ops and schedules a
redundant delete on a different replica; if both deletes land the container
falls below RF. Same safety consequence as the over-replication split-lock
delete, currently masked by the datanode-side term gate composed with the
5-minute RM startup wait exceeding the 30s heartbeat term propagation. MASKED:
classified by the consequence the defect would expose if the timing mask were
absent.
h3. Classification
* Verdict: MASKED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-ratis-replication
Guidance:
campaigns/ozone-9fbf9ee/targets/012-scm-ratis-replication/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-ratis-replication|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
Precondition: a CLOSED Ratis container (RF=3) with 4 replicas (over-replicated
by 1). The term-1 SCM leader schedules a delete for one replica; the delete is
recorded as a pending DELETE op AND dispatched to a datanode (in flight).
Action:
1. SCM leadership transfers. On becoming leader,
ReplicationManager.notifyStatusChanged calls
ContainerReplicaPendingOps.clear(), which wipes the pending-delete record. The
already-dispatched command is not recalled.
2. The new (term-2) leader re-derives over-replication from the current replica
view (still 4, the in-flight delete has not completed) with an empty
pending-ops list, so it schedules a second delete, on a possibly different
replica.
Observed (masked) outcome: if both deletes landed, the container would drop
from 4 to 2 replicas, below the replication factor. Under default configuration
this durable harm is masked: the datanode-side term gate drops the stale term-1
delete once the datanode learns the new SCM term (within about one heartbeat
interval, default 30s), and the new leader's post-leadership startup wait
(default 5m) far exceeds that, so the datanode has already dropped the stale
command before the new leader can schedule the competing one.
Unmasking condition: the durable below-replication-factor outcome becomes
reachable if the SCM startup wait after becoming leader is configured below the
datanode heartbeat term-propagation interval, so the new leader can dispatch
the second delete before the datanode has learned the new term and dropped the
stale one, and both deletes land. A plain unit test cannot inject the
datanode-side term-gate timing and selective command landing, so this patch
removes the SCM-side latent defect (the new leader losing track of the
in-flight delete) directly.
h3. Root cause
ReplicationManager.notifyStatusChanged
(hadoop-hdds/server-scm/.../ReplicationManager.java) clears the entire
ContainerReplicaPendingOps table on a leadership transition. This discards
in-flight DELETE ops, so the new leader has no record that a delete is already
in flight and can schedule a competing delete on a different replica. The
design relies entirely on the datanode term gate plus a timing composition
(startup wait much greater than heartbeat propagation) to prevent both deletes
from landing.
h3. Suggested fix
The attached patch [^HDDS-16449.001.patch] adds
ContainerReplicaPendingOps.clearExceptInFlightDeletes, which clears pending ADD
ops (and their scheduled-size accounting) but retains in-flight DELETE ops with
consistent counters, and calls it from notifyStatusChanged instead of clear().
The new leader therefore stays aware of deletes dispatched by the previous
leader and does not schedule a competing delete on a different replica; the
retained DELETE op is resent with the new term when it expires (via the
existing retained-delete resend path), and a container report still completes
it normally. This removes the SCM-side over-scheduling independent of the
datanode term-gate timing. Tests: a targeted unit test on
ContainerReplicaPendingOps asserts ADD ops are cleared while Ratis and EC
DELETE ops are retained with consistent counters, and the existing
TestReplicationManager leadership-transition test now asserts the in-flight
DELETE op is retained.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
On SCM leadership transfer the new leader clears pending ops and schedules a
redundant delete on a different replica; if both deletes land the container
falls below RF. Same safety consequence as the over-replication split-lock
delete, currently masked by the datanode-side term gate composed with the
5-minute RM startup wait exceeding the 30s heartbeat term propagation. MASKED:
classified by the consequence the defect would expose if the timing mask were
absent.
h3. Classification
* Verdict: MASKED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-ratis-replication
Guidance:
campaigns/ozone-9fbf9ee/targets/012-scm-ratis-replication/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-ratis-replication|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> SCM leadership transfer can schedule a redundant delete that drops a
> container below RF
> ---------------------------------------------------------------------------------------
>
> Key: HDDS-16449
> URL: https://issues.apache.org/jira/browse/HDDS-16449
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16449.001.patch
>
>
> h3. Finding
> On SCM leadership transfer the new leader clears pending ops and schedules a
> redundant delete on a different replica; if both deletes land the container
> falls below RF. Same safety consequence as the over-replication split-lock
> delete, currently masked by the datanode-side term gate composed with the
> 5-minute RM startup wait exceeding the 30s heartbeat term propagation.
> MASKED: classified by the consequence the defect would expose if the timing
> mask were absent.
> h3. Classification
> * Verdict: MASKED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: scm-ratis-replication
> Guidance:
> campaigns/ozone-9fbf9ee/targets/012-scm-ratis-replication/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-ratis-replication|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition: a CLOSED Ratis container (RF=3) with 4 replicas
> (over-replicated by 1). The term-1 SCM leader schedules a delete for one
> replica; the delete is recorded as a pending DELETE op AND dispatched to a
> datanode (in flight).
> Action:
> 1. SCM leadership transfers. On becoming leader,
> ReplicationManager.notifyStatusChanged calls
> ContainerReplicaPendingOps.clear(), which wipes the pending-delete record.
> The already-dispatched command is not recalled.
> 2. The new (term-2) leader re-derives over-replication from the current
> replica view (still 4, the in-flight delete has not completed) with an empty
> pending-ops list, so it schedules a second delete, on a possibly different
> replica.
> Observed (masked) outcome: if both deletes landed, the container would drop
> from 4 to 2 replicas, below the replication factor. Under default
> configuration this durable harm is masked: the datanode-side term gate drops
> the stale term-1 delete once the datanode learns the new SCM term (within
> about one heartbeat interval, default 30s), and the new leader's
> post-leadership startup wait (default 5m) far exceeds that, so the datanode
> has already dropped the stale command before the new leader can schedule the
> competing one.
> Unmasking condition: the durable below-replication-factor outcome becomes
> reachable if the SCM startup wait after becoming leader is configured below
> the datanode heartbeat term-propagation interval, so the new leader can
> dispatch the second delete before the datanode has learned the new term and
> dropped the stale one, and both deletes land. A plain unit test cannot inject
> the datanode-side term-gate timing and selective command landing, so this
> patch removes the SCM-side latent defect (the new leader losing track of the
> in-flight delete) directly.
> h3. Root cause
> ReplicationManager.notifyStatusChanged
> (hadoop-hdds/server-scm/.../ReplicationManager.java) clears the entire
> ContainerReplicaPendingOps table on a leadership transition. This discards
> in-flight DELETE ops, so the new leader has no record that a delete is
> already in flight and can schedule a competing delete on a different replica.
> The design relies entirely on the datanode term gate plus a timing
> composition (startup wait much greater than heartbeat propagation) to prevent
> both deletes from landing.
> h3. Suggested fix
> The attached patch [^HDDS-16449.001.patch] adds
> ContainerReplicaPendingOps.clearExceptInFlightDeletes, which clears pending
> ADD ops (and their scheduled-size accounting) but retains in-flight DELETE
> ops with consistent counters, and calls it from notifyStatusChanged instead
> of clear(). The new leader therefore stays aware of deletes dispatched by the
> previous leader and does not schedule a competing delete on a different
> replica; the retained DELETE op is resent with the new term when it expires
> (via the existing retained-delete resend path), and a container report still
> completes it normally. This removes the SCM-side over-scheduling independent
> of the datanode term-gate timing. Tests: a targeted unit test on
> ContainerReplicaPendingOps asserts ADD ops are cleared while Ratis and EC
> DELETE ops are retained with consistent counters, and the existing
> TestReplicationManager leadership-transition test now asserts the in-flight
> DELETE op is retained.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
