[
https://issues.apache.org/jira/browse/HDDS-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16457:
------------------------------
Description:
h3. Finding
An EC replica move plus a mid-move crash and restart leaves a durable two-copy
inconsistency (CLOSED source plus an UNHEALTHY destination) that no DN-local
mechanism reclaims, and the UNHEALTHY copy can win the in-memory slot and be
reported to SCM. Triggerable through normal DiskBalancer EC moves with a single
crash in the AtomicMove-to-import window; harm is a permanent orphaned replica
(disk leak plus volume-accounting drift) and a spurious UNHEALTHY report, so
bounded-but-durable rather than Critical.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-disk-balancer
Guidance:
campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-disk-balancer|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
1. An EC replica (replica index 1), CLOSED, resides on volume A. DiskBalancer
schedules a move to volume B.
2. The mover copies the replica to a tmp dir on B, stages the copied .container
as RECOVERING, and AtomicMoves the directory to B's final current/ location. It
has not yet run importContainer to reset the state back to CLOSED, and has not
yet marked the source for delete.
3. The datanode crashes in that window. Volume B now holds a RECOVERING EC
replica at its final location; volume A still holds the CLOSED source.
4. On restart, ContainerReader scans the dest volume first: the RECOVERING EC
replica is marked UNHEALTHY and added. Then it scans the source volume: the
CLOSED replica hits a duplicate, and resolveDuplicate for EC unconditionally
"leaves both containers on disk". The result is two on-disk authorities of one
replica index (CLOSED plus UNHEALTHY), the UNHEALTHY copy winning the in-memory
slot, with no datanode-local reconciliation.
h3. Root cause
The datanode move stages the destination replica at its final current/ location
before it commits, and restart reconciliation
({{ContainerReader.resolveDuplicate}}) has no rule for EC duplicates: it leaves
both copies on disk because EC replicas carry no BCSID and two copies may
legitimately have different replica indexes. It does not special-case the
mid-move leftover, where two copies share the same replica index and exactly
one is CLOSED.
h3. Suggested fix
Attached patch [^HDDS-16457.001.patch] extends {{resolveDuplicate}} so that,
for two EC copies of the SAME replica index where exactly one is CLOSED and the
other is not (RECOVERING/UNHEALTHY), the CLOSED copy wins and the non-CLOSED
duplicate is removed, mirroring the mismatched-state handling already used for
Ratis containers. Copies with different replica indexes, or any other state
combination, keep the existing leave-both behavior. A unit test drives the
crash-then-restart scan order through the real ContainerReader and asserts only
the CLOSED replica survives on disk and in memory.
Note: this fix reconciles the durable two-authority state at restart. Making
the destination replica non-authoritative until the move commits (for example
staging it where cleanupTmpDir reclaims it after a mid-move crash) would be a
complementary, more invasive hardening in the mover itself.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
An EC replica move plus a mid-move crash and restart leaves a durable two-copy
inconsistency (CLOSED source plus an UNHEALTHY destination) that no DN-local
mechanism reclaims, and the UNHEALTHY copy can win the in-memory slot and be
reported to SCM. Triggerable through normal DiskBalancer EC moves with a single
crash in the AtomicMove-to-import window; harm is a permanent orphaned replica
(disk leak plus volume-accounting drift) and a spurious UNHEALTHY report, so
bounded-but-durable rather than Critical.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-disk-balancer
Guidance:
campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-disk-balancer|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> DiskBalancer EC move plus crash leaves a durable CLOSED plus UNHEALTHY two
> copy inconsistency
> ---------------------------------------------------------------------------------------------
>
> Key: HDDS-16457
> URL: https://issues.apache.org/jira/browse/HDDS-16457
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16457.001.patch
>
>
> h3. Finding
> An EC replica move plus a mid-move crash and restart leaves a durable
> two-copy inconsistency (CLOSED source plus an UNHEALTHY destination) that no
> DN-local mechanism reclaims, and the UNHEALTHY copy can win the in-memory
> slot and be reported to SCM. Triggerable through normal DiskBalancer EC moves
> with a single crash in the AtomicMove-to-import window; harm is a permanent
> orphaned replica (disk leak plus volume-accounting drift) and a spurious
> UNHEALTHY report, so bounded-but-durable rather than Critical.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-disk-balancer
> Guidance:
> campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-disk-balancer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. An EC replica (replica index 1), CLOSED, resides on volume A. DiskBalancer
> schedules a move to volume B.
> 2. The mover copies the replica to a tmp dir on B, stages the copied
> .container as RECOVERING, and AtomicMoves the directory to B's final current/
> location. It has not yet run importContainer to reset the state back to
> CLOSED, and has not yet marked the source for delete.
> 3. The datanode crashes in that window. Volume B now holds a RECOVERING EC
> replica at its final location; volume A still holds the CLOSED source.
> 4. On restart, ContainerReader scans the dest volume first: the RECOVERING EC
> replica is marked UNHEALTHY and added. Then it scans the source volume: the
> CLOSED replica hits a duplicate, and resolveDuplicate for EC unconditionally
> "leaves both containers on disk". The result is two on-disk authorities of
> one replica index (CLOSED plus UNHEALTHY), the UNHEALTHY copy winning the
> in-memory slot, with no datanode-local reconciliation.
> h3. Root cause
> The datanode move stages the destination replica at its final current/
> location before it commits, and restart reconciliation
> ({{ContainerReader.resolveDuplicate}}) has no rule for EC duplicates: it
> leaves both copies on disk because EC replicas carry no BCSID and two copies
> may legitimately have different replica indexes. It does not special-case the
> mid-move leftover, where two copies share the same replica index and exactly
> one is CLOSED.
> h3. Suggested fix
> Attached patch [^HDDS-16457.001.patch] extends {{resolveDuplicate}} so that,
> for two EC copies of the SAME replica index where exactly one is CLOSED and
> the other is not (RECOVERING/UNHEALTHY), the CLOSED copy wins and the
> non-CLOSED duplicate is removed, mirroring the mismatched-state handling
> already used for Ratis containers. Copies with different replica indexes, or
> any other state combination, keep the existing leave-both behavior. A unit
> test drives the crash-then-restart scan order through the real
> ContainerReader and asserts only the CLOSED replica survives on disk and in
> memory.
> Note: this fix reconciles the durable two-authority state at restart. Making
> the destination replica non-authoritative until the move commits (for example
> staging it where cleanupTmpDir reclaims it after a mid-move crash) would be a
> complementary, more invasive hardening in the mover itself.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
