[
https://issues.apache.org/jira/browse/HDDS-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16450:
------------------------------
Description:
h3. Finding
Retained DELETE ops re-increment ReplicaDeleteTimeoutTotal /
EcReplicaDeleteTimeoutTotal on every scrubber cycle, so one logical delete
inflates the operator-facing JMX timeout metric N-fold; no safety impact
(subscribers are idempotent), only a permanently distorted monitoring signal,
so Low.
h3. Classification
* Verdict: REPRODUCED
* Severity: Low
* Source: Specula TLA+ model checking and confirmation debate, finding CR-3
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-pending-replica-ops
Guidance:
campaigns/ozone-9fbf9ee/targets/014-scm-pending-replica-ops/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-pending-replica-ops|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
Precondition: a DELETE replica op is scheduled via scheduleDeleteReplica and
its deadline passes without the datanode confirming the delete.
1. The periodic ExpiredContainerReplicaOpScrubber calls removeExpiredEntries().
The op is expired, so ReplicaDeleteTimeoutTotal (or
EcReplicaDeleteTimeoutTotal) is incremented and subscribers are notified with
timedOut=true. Unlike an ADD op, the DELETE op is intentionally kept in the map
(HDDS-12127: ReplicationManager must resend it).
2. On the next scrubber cycle the same unchanged op is still past its deadline,
so removeExpiredEntries increments the DELETE timeout metric again and
re-notifies subscribers.
3. This repeats once per scrubber cycle for as long as the op stays in the map,
so a single logical delete inflates the operator-facing JMX timeout counter
N-fold, while an ADD op (removed on first expiry) is counted exactly once.
Observed with a focused unit test: three scrubber cycles over one retained
DELETE produce ReplicaDeleteTimeoutTotal=3 (should be 1), whereas the ADD
control stays at 1.
h3. Root cause
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ContainerReplicaPendingOps.java:
in removeExpiredEntries, the DELETE branch keeps the op in the map
(HDDS-12127) but still falls through to expiredOps.add(op) and
updateTimeoutMetrics(op) on every pass. Because the op's deadline is unchanged,
each subsequent scrubber cycle re-observes it as expired and re-counts it.
updateTimeoutMetrics has been unchanged since the metric was introduced
(HDDS-6896) and was never reconsidered when retain-on-expiry was added. The
@Metric doc states the counter is "Number of container replicas which timed out
before being deleted" (per replica), so re-counting one replica violates the
documented semantics. No safety impact: the DELETE+timedOut subscribers are
idempotent (RM resends, SCMNodeManager ignores non-ADD, MoveManager treats an
already-gone move as a no-op); the only effect is the distorted metric.
h3. Suggested fix
Add a per-op timeoutCounted flag to ContainerReplicaOp. In
removeExpiredEntries, count and notify a retained DELETE op only the first time
it expires for its current deadline, then set the flag; ADD ops are unchanged
(removed on first expiry). A resend goes through addReplica, which builds a new
ContainerReplicaOp with the flag defaulting to false, so a distinct logical
delete is counted again on its own expiry. This preserves the retain-and-resend
design while making the DELETE timeout metric reflect distinct logical
operations. See [^HDDS-16450.001.patch].
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
Retained DELETE ops re-increment ReplicaDeleteTimeoutTotal /
EcReplicaDeleteTimeoutTotal on every scrubber cycle, so one logical delete
inflates the operator-facing JMX timeout metric N-fold; no safety impact
(subscribers are idempotent), only a permanently distorted monitoring signal,
so Low.
h3. Classification
* Verdict: REPRODUCED
* Severity: Low
* Source: Specula TLA+ model checking and confirmation debate, finding CR-3
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-pending-replica-ops
Guidance:
campaigns/ozone-9fbf9ee/targets/014-scm-pending-replica-ops/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-pending-replica-ops|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> Retained DELETE ops inflate ReplicaDeleteTimeout JMX metrics N-fold per
> logical delete
> --------------------------------------------------------------------------------------
>
> Key: HDDS-16450
> URL: https://issues.apache.org/jira/browse/HDDS-16450
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: HDDS-16450.001.patch
>
>
> h3. Finding
> Retained DELETE ops re-increment ReplicaDeleteTimeoutTotal /
> EcReplicaDeleteTimeoutTotal on every scrubber cycle, so one logical delete
> inflates the operator-facing JMX timeout metric N-fold; no safety impact
> (subscribers are idempotent), only a permanently distorted monitoring signal,
> so Low.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Low
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-3
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: scm-pending-replica-ops
> Guidance:
> campaigns/ozone-9fbf9ee/targets/014-scm-pending-replica-ops/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-pending-replica-ops|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition: a DELETE replica op is scheduled via scheduleDeleteReplica and
> its deadline passes without the datanode confirming the delete.
> 1. The periodic ExpiredContainerReplicaOpScrubber calls
> removeExpiredEntries(). The op is expired, so ReplicaDeleteTimeoutTotal (or
> EcReplicaDeleteTimeoutTotal) is incremented and subscribers are notified with
> timedOut=true. Unlike an ADD op, the DELETE op is intentionally kept in the
> map (HDDS-12127: ReplicationManager must resend it).
> 2. On the next scrubber cycle the same unchanged op is still past its
> deadline, so removeExpiredEntries increments the DELETE timeout metric again
> and re-notifies subscribers.
> 3. This repeats once per scrubber cycle for as long as the op stays in the
> map, so a single logical delete inflates the operator-facing JMX timeout
> counter N-fold, while an ADD op (removed on first expiry) is counted exactly
> once.
> Observed with a focused unit test: three scrubber cycles over one retained
> DELETE produce ReplicaDeleteTimeoutTotal=3 (should be 1), whereas the ADD
> control stays at 1.
> h3. Root cause
> hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/replication/ContainerReplicaPendingOps.java:
> in removeExpiredEntries, the DELETE branch keeps the op in the map
> (HDDS-12127) but still falls through to expiredOps.add(op) and
> updateTimeoutMetrics(op) on every pass. Because the op's deadline is
> unchanged, each subsequent scrubber cycle re-observes it as expired and
> re-counts it. updateTimeoutMetrics has been unchanged since the metric was
> introduced (HDDS-6896) and was never reconsidered when retain-on-expiry was
> added. The @Metric doc states the counter is "Number of container replicas
> which timed out before being deleted" (per replica), so re-counting one
> replica violates the documented semantics. No safety impact: the
> DELETE+timedOut subscribers are idempotent (RM resends, SCMNodeManager
> ignores non-ADD, MoveManager treats an already-gone move as a no-op); the
> only effect is the distorted metric.
> h3. Suggested fix
> Add a per-op timeoutCounted flag to ContainerReplicaOp. In
> removeExpiredEntries, count and notify a retained DELETE op only the first
> time it expires for its current deadline, then set the flag; ADD ops are
> unchanged (removed on first expiry). A resend goes through addReplica, which
> builds a new ContainerReplicaOp with the flag defaulting to false, so a
> distinct logical delete is counted again on its own expiry. This preserves
> the retain-and-resend design while making the DELETE timeout metric reflect
> distinct logical operations. See [^HDDS-16450.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
