[
https://issues.apache.org/jira/browse/HDDS-16455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16455:
------------------------------
Description:
h3. Finding
A stale scrub task silently demotes a successfully-CLOSED reconstructed EC
replica to UNHEALTHY with no ICR; on the next full container report SCM deletes
and re-reconstructs a valid replica. Reachable via the normal scrub background
thread racing the reconstruction/close path; harm is persistent on the datanode
but bounded (SCM re-replicates), so High rather than Critical.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-ec-reconstruction
Guidance:
campaigns/ozone-9fbf9ee/targets/022-dn-ec-reconstruction/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-ec-reconstruction|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
1. An EC reconstruction target is created RECOVERING; its
{{recoveringContainerMap}} entry carries a deadline (create time plus
recovering timeout).
2. Reconstruction stalls past the deadline, so
{{StaleRecoveringContainerScrubbingService.getTasks}} enqueues a
{{RecoveringContainerScrubbingTask}} and removes the map entry, all while the
container is still RECOVERING.
3. Reconstruction then completes: the replica is moved RECOVERING to CLOSING to
CLOSED through the normal close path.
4. The queued scrub task fires on the background pool. It re-checks only the
deadline (now null, so it proceeds) and calls {{markContainerUnhealthy()}},
which has no state guard, so the successfully CLOSED replica is silently
demoted to UNHEALTHY. No ICR is sent, so SCM only learns of the UNHEALTHY
replica on the next full container report and then deletes and re-reconstructs
a valid replica.
h3. Root cause
{{StaleRecoveringContainerScrubbingService}} splits its "decide stale" step
({{getTasks}}) from its "act" step ({{RecoveringContainerScrubbingTask.call}})
across two threads with no state re-check between them. {{call()}} re-reads
only the deadline (added earlier for the deadline-extension race) and never
re-checks the container state before calling
{{Container.markContainerUnhealthy}}.
{{KeyValueContainer.markContainerUnhealthy}} unconditionally sets state to
UNHEALTHY, so a container that reached a terminal CLOSED (or CLOSING) state
between scheduling and execution is degraded anyway.
h3. Suggested fix
Attached patch [^HDDS-16455.001.patch] re-checks the container state under the
container write lock inside {{RecoveringContainerScrubbingTask.call}} and marks
the container UNHEALTHY only if it is still RECOVERING, mirroring the
write-lock discipline that {{KeyValueHandler.markContainerForClose}} already
uses. A container that has moved out of RECOVERING (closed or deleted) is left
untouched. A unit test in {{TestStaleRecoveringContainerScrubbingService}}
drives the schedule then close then execute interleaving and asserts the
replica stays CLOSED.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
A stale scrub task silently demotes a successfully-CLOSED reconstructed EC
replica to UNHEALTHY with no ICR; on the next full container report SCM deletes
and re-reconstructs a valid replica. Reachable via the normal scrub background
thread racing the reconstruction/close path; harm is persistent on the datanode
but bounded (SCM re-replicates), so High rather than Critical.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-ec-reconstruction
Guidance:
campaigns/ozone-9fbf9ee/targets/022-dn-ec-reconstruction/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-ec-reconstruction|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> Stale scrub task demotes a CLOSED reconstructed EC replica to UNHEALTHY,
> forcing SCM re reconstruction
> ------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16455
> URL: https://issues.apache.org/jira/browse/HDDS-16455
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16455.001.patch
>
>
> h3. Finding
> A stale scrub task silently demotes a successfully-CLOSED reconstructed EC
> replica to UNHEALTHY with no ICR; on the next full container report SCM
> deletes and re-reconstructs a valid replica. Reachable via the normal scrub
> background thread racing the reconstruction/close path; harm is persistent on
> the datanode but bounded (SCM re-replicates), so High rather than Critical.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-ec-reconstruction
> Guidance:
> campaigns/ozone-9fbf9ee/targets/022-dn-ec-reconstruction/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-ec-reconstruction|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. An EC reconstruction target is created RECOVERING; its
> {{recoveringContainerMap}} entry carries a deadline (create time plus
> recovering timeout).
> 2. Reconstruction stalls past the deadline, so
> {{StaleRecoveringContainerScrubbingService.getTasks}} enqueues a
> {{RecoveringContainerScrubbingTask}} and removes the map entry, all while the
> container is still RECOVERING.
> 3. Reconstruction then completes: the replica is moved RECOVERING to CLOSING
> to CLOSED through the normal close path.
> 4. The queued scrub task fires on the background pool. It re-checks only the
> deadline (now null, so it proceeds) and calls {{markContainerUnhealthy()}},
> which has no state guard, so the successfully CLOSED replica is silently
> demoted to UNHEALTHY. No ICR is sent, so SCM only learns of the UNHEALTHY
> replica on the next full container report and then deletes and
> re-reconstructs a valid replica.
> h3. Root cause
> {{StaleRecoveringContainerScrubbingService}} splits its "decide stale" step
> ({{getTasks}}) from its "act" step
> ({{RecoveringContainerScrubbingTask.call}}) across two threads with no state
> re-check between them. {{call()}} re-reads only the deadline (added earlier
> for the deadline-extension race) and never re-checks the container state
> before calling {{Container.markContainerUnhealthy}}.
> {{KeyValueContainer.markContainerUnhealthy}} unconditionally sets state to
> UNHEALTHY, so a container that reached a terminal CLOSED (or CLOSING) state
> between scheduling and execution is degraded anyway.
> h3. Suggested fix
> Attached patch [^HDDS-16455.001.patch] re-checks the container state under
> the container write lock inside {{RecoveringContainerScrubbingTask.call}} and
> marks the container UNHEALTHY only if it is still RECOVERING, mirroring the
> write-lock discipline that {{KeyValueHandler.markContainerForClose}} already
> uses. A container that has moved out of RECOVERING (closed or deleted) is
> left untouched. A unit test in
> {{TestStaleRecoveringContainerScrubbingService}} drives the schedule then
> close then execute interleaving and asserts the replica stays CLOSED.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
