[
https://issues.apache.org/jira/browse/HDDS-16457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishna Kumar Asawa reassigned HDDS-16457:
------------------------------------------
Assignee: Gargi Jaiswal
> DiskBalancer EC move plus crash leaves a durable CLOSED plus UNHEALTHY two
> copy inconsistency
> ---------------------------------------------------------------------------------------------
>
> Key: HDDS-16457
> URL: https://issues.apache.org/jira/browse/HDDS-16457
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Gargi Jaiswal
> Priority: Major
> Attachments: HDDS-16457.001.patch
>
>
> h3. Finding
> An EC replica move plus a mid-move crash and restart leaves a durable
> two-copy inconsistency (CLOSED source plus an UNHEALTHY destination) that no
> DN-local mechanism reclaims, and the UNHEALTHY copy can win the in-memory
> slot and be reported to SCM. Triggerable through normal DiskBalancer EC moves
> with a single crash in the AtomicMove-to-import window; harm is a permanent
> orphaned replica (disk leak plus volume-accounting drift) and a spurious
> UNHEALTHY report, so bounded-but-durable rather than Critical.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-disk-balancer
> Guidance:
> campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-disk-balancer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. An EC replica (replica index 1), CLOSED, resides on volume A. DiskBalancer
> schedules a move to volume B.
> 2. The mover copies the replica to a tmp dir on B, stages the copied
> .container as RECOVERING, and AtomicMoves the directory to B's final current/
> location. It has not yet run importContainer to reset the state back to
> CLOSED, and has not yet marked the source for delete.
> 3. The datanode crashes in that window. Volume B now holds a RECOVERING EC
> replica at its final location; volume A still holds the CLOSED source.
> 4. On restart, ContainerReader scans the dest volume first: the RECOVERING EC
> replica is marked UNHEALTHY and added. Then it scans the source volume: the
> CLOSED replica hits a duplicate, and resolveDuplicate for EC unconditionally
> "leaves both containers on disk". The result is two on-disk authorities of
> one replica index (CLOSED plus UNHEALTHY), the UNHEALTHY copy winning the
> in-memory slot, with no datanode-local reconciliation.
> h3. Root cause
> The datanode move stages the destination replica at its final current/
> location before it commits, and restart reconciliation
> ({{ContainerReader.resolveDuplicate}}) has no rule for EC duplicates: it
> leaves both copies on disk because EC replicas carry no BCSID and two copies
> may legitimately have different replica indexes. It does not special-case the
> mid-move leftover, where two copies share the same replica index and exactly
> one is CLOSED.
> h3. Suggested fix
> Attached patch [^HDDS-16457.001.patch] extends {{resolveDuplicate}} so that,
> for two EC copies of the SAME replica index where exactly one is CLOSED and
> the other is not (RECOVERING/UNHEALTHY), the CLOSED copy wins and the
> non-CLOSED duplicate is removed, mirroring the mismatched-state handling
> already used for Ratis containers. Copies with different replica indexes, or
> any other state combination, keep the existing leave-both behavior. A unit
> test drives the crash-then-restart scan order through the real
> ContainerReader and asserts only the CLOSED replica survives on disk and in
> memory.
> Note: this fix reconciles the durable two-authority state at restart. Making
> the destination replica non-authoritative until the move commits (for example
> staging it where cleanupTmpDir reclaims it after a mid-move crash) would be a
> complementary, more invasive hardening in the mover itself.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
