[
https://issues.apache.org/jira/browse/HDDS-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132342#comment-18132342
]
Gargi Jaiswal commented on HDDS-16458:
--------------------------------------
[~smeng] thanks for identifying the issue.
This issue was already known to us that when there is a crashing in between
deleting old container replica after diskbalancer move can cause duplicate
containers on disk, leaving one replica as orphan.
Internally we had discussion about this case of resolving duplicate EC
container is quite risky.
- First this needs to be handled like it is done for RATIS duplicated
containers as duplicate EC containers can be there for any reasons irrespective
of diskbalancer.
- EC duplicates can't be resolved like Ratis duplicates. Ratis uses the BCSID,
but EC writes have BCSID 0, so there is no way to tell which copy is newer or
more complete. With different replica indexes, the copies hold different data,
so deleting either one loses part of the stripe.
[~erose] in case of diskbalancer where both old and new container replica
would have same replica index would it be safe to delete the old replica so
that after dn crash there is no duplicate EC containers?
cc: [~Sammi]
> DiskBalancer EC move crash before source delete mark leaves a permanent
> orphaned replica
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-16458
> URL: https://issues.apache.org/jira/browse/HDDS-16458
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Gargi Jaiswal
> Priority: Major
> Attachments: HDDS-16458.001.patch
>
>
> h3. Finding
> After a DiskBalancer EC move, a crash (or swallowed mark failure) before the
> source is durably marked for delete drops the volatile pending-deletion
> queue, and restart reconciliation deliberately leaves both EC copies,
> producing a permanent orphaned replica untracked in memory. Triggerable via
> normal EC moves plus one crash; harm is a durable disk leak and under-counted
> volume accounting, bounded and not client-visible corruption.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-disk-balancer
> Guidance:
> campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-disk-balancer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. DiskBalancer selects a CLOSED EC replica for a move (there is no EC
> exclusion in the choosing policy).
> 2. The move imports the destination copy (CLOSED) and swaps the in-memory
> authority to it via updateContainer.
> 3. markContainerForDelete on the source fails and the exception is swallowed:
> the source's on-disk .container stays CLOSED, and the source is only enqueued
> in the volatile, non-persistent pendingDeletionContainers with a deferred
> deadline. The move is still reported successful.
> 4. The datanode crashes before the deadline, dropping the volatile queue with
> no durable record.
> 5. On restart, ContainerReader.resolveDuplicate for EC "leaves both
> containers on disk", so both CLOSED copies of the same replica index persist
> forever. Nothing reclaims the source. (Ratis self-heals via BCSID/state; EC
> does not.)
> h3. Root cause
> When markContainerForDelete fails, the source's DELETED intent is neither
> persisted (the mark, which writes it to the .container file, failed) nor
> durably recorded elsewhere: it lives only in the in-memory
> pendingDeletionContainers queue for the deferred-deletion window. A crash in
> that window loses it, and EC restart reconciliation does not remove a
> same-index CLOSED duplicate, so the source becomes a permanent orphan. The
> failure is swallowed with a comment that restart will handle it, which is not
> true for EC.
> h3. Suggested fix
> Attached patch [^HDDS-16458.001.patch] deletes the source replica
> synchronously when markContainerForDelete fails, instead of relying on the
> volatile queue plus a restart. At that point the imported destination is
> already the in-memory authority, so the source is non-authoritative and can
> be removed immediately via the existing deletion path; it is not also
> enqueued for deferred deletion. This removes the durable-duplicate window
> without depending on a persisted DELETED state or on restart reconciliation.
> A unit test drives a real DiskBalancer move with a failing source mark, then
> models a crash and restart, and asserts exactly one copy survives (and that
> the source is not left in the pending-deletion queue).
> Note: a crash landing between the in-memory authority swap and this
> synchronous cleanup is a narrower residual window; fully closing it would
> require persisting the source's delete intent durably at move-commit time (a
> larger change in the move protocol).
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
