[
https://issues.apache.org/jira/browse/HDDS-16458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16458:
------------------------------
Description:
h3. Finding
After a DiskBalancer EC move, a crash (or swallowed mark failure) before the
source is durably marked for delete drops the volatile pending-deletion queue,
and restart reconciliation deliberately leaves both EC copies, producing a
permanent orphaned replica untracked in memory. Triggerable via normal EC moves
plus one crash; harm is a durable disk leak and under-counted volume
accounting, bounded and not client-visible corruption.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-disk-balancer
Guidance:
campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-disk-balancer|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
1. DiskBalancer selects a CLOSED EC replica for a move (there is no EC
exclusion in the choosing policy).
2. The move imports the destination copy (CLOSED) and swaps the in-memory
authority to it via updateContainer.
3. markContainerForDelete on the source fails and the exception is swallowed:
the source's on-disk .container stays CLOSED, and the source is only enqueued
in the volatile, non-persistent pendingDeletionContainers with a deferred
deadline. The move is still reported successful.
4. The datanode crashes before the deadline, dropping the volatile queue with
no durable record.
5. On restart, ContainerReader.resolveDuplicate for EC "leaves both containers
on disk", so both CLOSED copies of the same replica index persist forever.
Nothing reclaims the source. (Ratis self-heals via BCSID/state; EC does not.)
h3. Root cause
When markContainerForDelete fails, the source's DELETED intent is neither
persisted (the mark, which writes it to the .container file, failed) nor
durably recorded elsewhere: it lives only in the in-memory
pendingDeletionContainers queue for the deferred-deletion window. A crash in
that window loses it, and EC restart reconciliation does not remove a
same-index CLOSED duplicate, so the source becomes a permanent orphan. The
failure is swallowed with a comment that restart will handle it, which is not
true for EC.
h3. Suggested fix
Attached patch [^HDDS-16458.001.patch] deletes the source replica synchronously
when markContainerForDelete fails, instead of relying on the volatile queue
plus a restart. At that point the imported destination is already the in-memory
authority, so the source is non-authoritative and can be removed immediately
via the existing deletion path; it is not also enqueued for deferred deletion.
This removes the durable-duplicate window without depending on a persisted
DELETED state or on restart reconciliation. A unit test drives a real
DiskBalancer move with a failing source mark, then models a crash and restart,
and asserts exactly one copy survives (and that the source is not left in the
pending-deletion queue).
Note: a crash landing between the in-memory authority swap and this synchronous
cleanup is a narrower residual window; fully closing it would require
persisting the source's delete intent durably at move-commit time (a larger
change in the move protocol).
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
After a DiskBalancer EC move, a crash (or swallowed mark failure) before the
source is durably marked for delete drops the volatile pending-deletion queue,
and restart reconciliation deliberately leaves both EC copies, producing a
permanent orphaned replica untracked in memory. Triggerable via normal EC moves
plus one crash; harm is a durable disk leak and under-counted volume
accounting, bounded and not client-visible corruption.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-disk-balancer
Guidance:
campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-disk-balancer|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> DiskBalancer EC move crash before source delete mark leaves a permanent
> orphaned replica
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-16458
> URL: https://issues.apache.org/jira/browse/HDDS-16458
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16458.001.patch
>
>
> h3. Finding
> After a DiskBalancer EC move, a crash (or swallowed mark failure) before the
> source is durably marked for delete drops the volatile pending-deletion
> queue, and restart reconciliation deliberately leaves both EC copies,
> producing a permanent orphaned replica untracked in memory. Triggerable via
> normal EC moves plus one crash; harm is a durable disk leak and under-counted
> volume accounting, bounded and not client-visible corruption.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-disk-balancer
> Guidance:
> campaigns/ozone-9fbf9ee/targets/021-dn-disk-balancer/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-disk-balancer|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. DiskBalancer selects a CLOSED EC replica for a move (there is no EC
> exclusion in the choosing policy).
> 2. The move imports the destination copy (CLOSED) and swaps the in-memory
> authority to it via updateContainer.
> 3. markContainerForDelete on the source fails and the exception is swallowed:
> the source's on-disk .container stays CLOSED, and the source is only enqueued
> in the volatile, non-persistent pendingDeletionContainers with a deferred
> deadline. The move is still reported successful.
> 4. The datanode crashes before the deadline, dropping the volatile queue with
> no durable record.
> 5. On restart, ContainerReader.resolveDuplicate for EC "leaves both
> containers on disk", so both CLOSED copies of the same replica index persist
> forever. Nothing reclaims the source. (Ratis self-heals via BCSID/state; EC
> does not.)
> h3. Root cause
> When markContainerForDelete fails, the source's DELETED intent is neither
> persisted (the mark, which writes it to the .container file, failed) nor
> durably recorded elsewhere: it lives only in the in-memory
> pendingDeletionContainers queue for the deferred-deletion window. A crash in
> that window loses it, and EC restart reconciliation does not remove a
> same-index CLOSED duplicate, so the source becomes a permanent orphan. The
> failure is swallowed with a comment that restart will handle it, which is not
> true for EC.
> h3. Suggested fix
> Attached patch [^HDDS-16458.001.patch] deletes the source replica
> synchronously when markContainerForDelete fails, instead of relying on the
> volatile queue plus a restart. At that point the imported destination is
> already the in-memory authority, so the source is non-authoritative and can
> be removed immediately via the existing deletion path; it is not also
> enqueued for deferred deletion. This removes the durable-duplicate window
> without depending on a persisted DELETED state or on restart reconciliation.
> A unit test drives a real DiskBalancer move with a failing source mark, then
> models a crash and restart, and asserts exactly one copy survives (and that
> the source is not left in the pending-deletion queue).
> Note: a crash landing between the in-memory authority swap and this
> synchronous cleanup is a narrower residual window; fully closing it would
> require persisting the source's delete intent durably at move-commit time (a
> larger change in the move protocol).
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
