[
https://issues.apache.org/jira/browse/HDDS-16447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16447:
------------------------------
Description:
h3. Finding
An out-of-lock race re-inserts a DELETE op for an already-deleted replica,
producing an orphaned pending op that is never confirmable and resent every RM
cycle; the inflated inflight-delete count propagates to a real external effect
(RM issues spurious replicate and redundant delete commands to datanodes, and
over/under-replication accounting stays corrupted for the whole SCM leadership
term). Persistent until leadership change, so High rather than Critical.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-pending-replica-ops
Guidance:
campaigns/ozone-9fbf9ee/targets/014-scm-pending-replica-ops/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-pending-replica-ops|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
Precondition: a container has a pending DELETE op for a replica on datanode d
that has passed its deadline. Two SCM threads run concurrently: the
ReplicationManager monitor (ContainerReplicaPendingOps.removeExpiredEntries)
and the container-report / ICR handler
(ContainerStateManagerImpl.removeContainerReplica).
Action / interleaving:
1. removeExpiredEntries finds the expired DELETE op. DELETE ops are
intentionally retained in the list (RM must resend them, or they are removed
when a report confirms the deletion), so the op stays in the list; the
container write lock is then released and subscribers are notified out of the
lock with timedOut=true.
2. In that out-of-lock window a container report confirming the replica is gone
runs removeContainerReplica: it removes the replica from the authoritative map
AND calls completeDeleteReplica, which removes the still-present DELETE op.
3. The out-of-lock notification reaches ReplicationManager.opCompleted, which
resends the delete via sendDatanodeCommand, and scheduleDeleteReplica
re-inserts a fresh DELETE op.
Observed wrong outcome: a pending DELETE op now exists for a replica already
removed from the authoritative map. No future container report can complete it
(completeDeleteReplica only fires on a present to absent transition, and the
replica is already absent), and removeExpiredEntries never removes DELETE ops,
so the orphan op is resent every RM cycle forever. This keeps
getPendingOpCount(DELETE) permanently inflated and sends redundant delete
commands for a non-existent replica for the whole SCM leadership term (clear()
on leadership change resets it).
h3. Root cause
ReplicationManager.opCompleted
(hadoop-hdds/server-scm/.../ReplicationManager.java) resends an expired DELETE
unconditionally, without checking whether the target replica is still tracked.
Because the retained-DELETE resend notification is delivered out of the
container lock (ContainerReplicaPendingOps.removeExpiredEntries), a concurrent
report-driven removeContainerReplica can remove both the replica and the
pending op in the window before the resend runs, and the resend re-inserts an
orphan op via scheduleDeleteReplica / addReplica (which re-adds
unconditionally).
h3. Suggested fix
The attached patch [^HDDS-16447.001.patch] makes ReplicationManager.opCompleted
resend the expired delete only when the target replica (same datanode and
replica index) is still present in the authoritative container replica map. If
the replica has already been removed (the deletion was confirmed by a report),
the resend is skipped, so no orphan pending DELETE op is created. Two tests are
added to TestReplicationManager: one asserts the delete is still resent and
re-tracked when the replica is present, and one asserts no command is sent and
no pending DELETE op is re-inserted when the replica is already gone.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
An out-of-lock race re-inserts a DELETE op for an already-deleted replica,
producing an orphaned pending op that is never confirmable and resent every RM
cycle; the inflated inflight-delete count propagates to a real external effect
(RM issues spurious replicate and redundant delete commands to datanodes, and
over/under-replication accounting stays corrupted for the whole SCM leadership
term). Persistent until leadership change, so High rather than Critical.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-pending-replica-ops
Guidance:
campaigns/ozone-9fbf9ee/targets/014-scm-pending-replica-ops/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-pending-replica-ops|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> SCM out of lock race re inserts a DELETE op for an already deleted replica,
> corrupting inflight accounting
> ----------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16447
> URL: https://issues.apache.org/jira/browse/HDDS-16447
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16447.001.patch
>
>
> h3. Finding
> An out-of-lock race re-inserts a DELETE op for an already-deleted replica,
> producing an orphaned pending op that is never confirmable and resent every
> RM cycle; the inflated inflight-delete count propagates to a real external
> effect (RM issues spurious replicate and redundant delete commands to
> datanodes, and over/under-replication accounting stays corrupted for the
> whole SCM leadership term). Persistent until leadership change, so High
> rather than Critical.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: scm-pending-replica-ops
> Guidance:
> campaigns/ozone-9fbf9ee/targets/014-scm-pending-replica-ops/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-pending-replica-ops|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition: a container has a pending DELETE op for a replica on datanode d
> that has passed its deadline. Two SCM threads run concurrently: the
> ReplicationManager monitor (ContainerReplicaPendingOps.removeExpiredEntries)
> and the container-report / ICR handler
> (ContainerStateManagerImpl.removeContainerReplica).
> Action / interleaving:
> 1. removeExpiredEntries finds the expired DELETE op. DELETE ops are
> intentionally retained in the list (RM must resend them, or they are removed
> when a report confirms the deletion), so the op stays in the list; the
> container write lock is then released and subscribers are notified out of the
> lock with timedOut=true.
> 2. In that out-of-lock window a container report confirming the replica is
> gone runs removeContainerReplica: it removes the replica from the
> authoritative map AND calls completeDeleteReplica, which removes the
> still-present DELETE op.
> 3. The out-of-lock notification reaches ReplicationManager.opCompleted, which
> resends the delete via sendDatanodeCommand, and scheduleDeleteReplica
> re-inserts a fresh DELETE op.
> Observed wrong outcome: a pending DELETE op now exists for a replica already
> removed from the authoritative map. No future container report can complete
> it (completeDeleteReplica only fires on a present to absent transition, and
> the replica is already absent), and removeExpiredEntries never removes DELETE
> ops, so the orphan op is resent every RM cycle forever. This keeps
> getPendingOpCount(DELETE) permanently inflated and sends redundant delete
> commands for a non-existent replica for the whole SCM leadership term
> (clear() on leadership change resets it).
> h3. Root cause
> ReplicationManager.opCompleted
> (hadoop-hdds/server-scm/.../ReplicationManager.java) resends an expired
> DELETE unconditionally, without checking whether the target replica is still
> tracked. Because the retained-DELETE resend notification is delivered out of
> the container lock (ContainerReplicaPendingOps.removeExpiredEntries), a
> concurrent report-driven removeContainerReplica can remove both the replica
> and the pending op in the window before the resend runs, and the resend
> re-inserts an orphan op via scheduleDeleteReplica / addReplica (which re-adds
> unconditionally).
> h3. Suggested fix
> The attached patch [^HDDS-16447.001.patch] makes
> ReplicationManager.opCompleted resend the expired delete only when the target
> replica (same datanode and replica index) is still present in the
> authoritative container replica map. If the replica has already been removed
> (the deletion was confirmed by a report), the resend is skipped, so no orphan
> pending DELETE op is created. Two tests are added to TestReplicationManager:
> one asserts the delete is still resent and re-tracked when the replica is
> present, and one asserts no command is sent and no pending DELETE op is
> re-inserted when the replica is already gone.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
