[
https://issues.apache.org/jira/browse/HDDS-16461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16461:
------------------------------
Description:
h3. Finding
An asymmetric decrement in the V2/V3 block-deletion path leaves the persisted
pendingDeleteBlockCount inflated on a fully drained container (an internal
invariant violation reachable via the real mark path plus one background
deletion pass when a block is processed more times than it is unlinked, e.g.
duplicate txns per HDDS-11267 or SCM at-least-once redelivery). No live
consumer reads the persisted value while the DN runs; the only activation path
(restart reload) is masked by resetPendingDeleteBlockCount (HDDS-7156), which
converts the divergence into one wasted deletion pass and a log line rather
than external corruption.
h3. Classification
* Verdict: MASKED
* Severity: Medium
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-block-deletion
Guidance:
campaigns/ozone-9fbf9ee/targets/017-dn-block-deletion/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-block-deletion|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
1. Under SCHEMA_V2/SCHEMA_V3, two distinct delete transactions (txid=1 and
txid=2) reference the same block b1 (a handled production case:
skip-duplicate-deletion, plus SCM at-least-once redelivery). Both are marked,
so the persisted and in-memory pendingDeleteBlockCount each reach 2.
2. One background deletion pass physically unlinks b1 once (blocksDeleted=1)
but processes it twice (blocksProcessed=2). The in-memory counter is
decremented by blocksProcessed (2) and clamped at 0, so it drains to 0. The
persisted DB counter is decremented by blocksDeleted (1) with no floor, so it
is left at 1.
3. The container is now fully drained (no delete transactions, nothing in
flight) but the persisted pendingDeleteBlockCount is 1, violating the
drained-consistency invariant.
Masked consequence (why this is a latent defect, not a live bug at HEAD): no
live consumer reads the persisted value while the datanode runs (the in-memory
value, which is correct, drives selection and reports). The only activation
path is a restart, which reloads the inflated persisted value into memory and
re-selects the drained container for one wasted deletion pass; that pass finds
no transactions and calls resetPendingDeleteBlockCount, which converts the
divergence into a "Pending block deletion not found" log rather than external
corruption. This is the long-standing symptom tracked publicly as HDDS-8063
(open, unresolved); the reset was a workaround (HDDS-7156), not a root-cause
fix.
h3. Root cause
The decrement of the pending-deletion counters is asymmetric between memory and
disk. In {{BlockDeletingTask.deleteViaTransactionStore}} the in-memory
{{ContainerData.decDeletion}} uses the processed-occurrence counts
(blocksProcessed / bytesProcessed) and floors at 0, while
{{KeyValueContainerData.updateAndCommitDBCounters}} decrements the persisted
pending count and bytes by the deduplicated deleted-block count and released
bytes, with no floor. Whenever blocksProcessed exceeds blocksDeleted the
persisted counters are left inflated on a drained container.
h3. Suggested fix
Attached patch [^HDDS-16461.001.patch] makes the persisted decrement symmetric
with the in-memory one: {{updateAndCommitDBCounters}} takes explicit
pending-deletion count and bytes decrement operands and floors both at 0
(mirroring {{ContainerData.decDeletion}}). The V2/V3 caller passes the
processed-occurrence counts (blocksProcessed / bytesProcessed); the V1 caller
passes its deduplicated counts unchanged, since V1 processes each block once. A
unit test in {{TestBlockDeletingService}} marks two delete transactions for the
same block, runs one deletion pass, and asserts the persisted pending count and
bytes both drain to 0. This addresses the root cause of the publicly tracked
HDDS-8063 and reduces the {{resetPendingDeleteBlockCount}} workaround to a
safety net.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
An asymmetric decrement in the V2/V3 block-deletion path leaves the persisted
pendingDeleteBlockCount inflated on a fully drained container (an internal
invariant violation reachable via the real mark path plus one background
deletion pass when a block is processed more times than it is unlinked, e.g.
duplicate txns per HDDS-11267 or SCM at-least-once redelivery). No live
consumer reads the persisted value while the DN runs; the only activation path
(restart reload) is masked by resetPendingDeleteBlockCount (HDDS-7156), which
converts the divergence into one wasted deletion pass and a log line rather
than external corruption.
h3. Classification
* Verdict: MASKED
* Severity: Medium
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-block-deletion
Guidance:
campaigns/ozone-9fbf9ee/targets/017-dn-block-deletion/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-block-deletion|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> Asymmetric decrement inflates persisted pendingDeleteBlockCount (masked by
> restart reset)
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-16461
> URL: https://issues.apache.org/jira/browse/HDDS-16461
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: HDDS-16461.001.patch
>
>
> h3. Finding
> An asymmetric decrement in the V2/V3 block-deletion path leaves the persisted
> pendingDeleteBlockCount inflated on a fully drained container (an internal
> invariant violation reachable via the real mark path plus one background
> deletion pass when a block is processed more times than it is unlinked, e.g.
> duplicate txns per HDDS-11267 or SCM at-least-once redelivery). No live
> consumer reads the persisted value while the DN runs; the only activation
> path (restart reload) is masked by resetPendingDeleteBlockCount (HDDS-7156),
> which converts the divergence into one wasted deletion pass and a log line
> rather than external corruption.
> h3. Classification
> * Verdict: MASKED
> * Severity: Medium
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: dn-block-deletion
> Guidance:
> campaigns/ozone-9fbf9ee/targets/017-dn-block-deletion/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "dn-block-deletion|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. Under SCHEMA_V2/SCHEMA_V3, two distinct delete transactions (txid=1 and
> txid=2) reference the same block b1 (a handled production case:
> skip-duplicate-deletion, plus SCM at-least-once redelivery). Both are marked,
> so the persisted and in-memory pendingDeleteBlockCount each reach 2.
> 2. One background deletion pass physically unlinks b1 once (blocksDeleted=1)
> but processes it twice (blocksProcessed=2). The in-memory counter is
> decremented by blocksProcessed (2) and clamped at 0, so it drains to 0. The
> persisted DB counter is decremented by blocksDeleted (1) with no floor, so it
> is left at 1.
> 3. The container is now fully drained (no delete transactions, nothing in
> flight) but the persisted pendingDeleteBlockCount is 1, violating the
> drained-consistency invariant.
> Masked consequence (why this is a latent defect, not a live bug at HEAD): no
> live consumer reads the persisted value while the datanode runs (the
> in-memory value, which is correct, drives selection and reports). The only
> activation path is a restart, which reloads the inflated persisted value into
> memory and re-selects the drained container for one wasted deletion pass;
> that pass finds no transactions and calls resetPendingDeleteBlockCount, which
> converts the divergence into a "Pending block deletion not found" log rather
> than external corruption. This is the long-standing symptom tracked publicly
> as HDDS-8063 (open, unresolved); the reset was a workaround (HDDS-7156), not
> a root-cause fix.
> h3. Root cause
> The decrement of the pending-deletion counters is asymmetric between memory
> and disk. In {{BlockDeletingTask.deleteViaTransactionStore}} the in-memory
> {{ContainerData.decDeletion}} uses the processed-occurrence counts
> (blocksProcessed / bytesProcessed) and floors at 0, while
> {{KeyValueContainerData.updateAndCommitDBCounters}} decrements the persisted
> pending count and bytes by the deduplicated deleted-block count and released
> bytes, with no floor. Whenever blocksProcessed exceeds blocksDeleted the
> persisted counters are left inflated on a drained container.
> h3. Suggested fix
> Attached patch [^HDDS-16461.001.patch] makes the persisted decrement
> symmetric with the in-memory one: {{updateAndCommitDBCounters}} takes
> explicit pending-deletion count and bytes decrement operands and floors both
> at 0 (mirroring {{ContainerData.decDeletion}}). The V2/V3 caller passes the
> processed-occurrence counts (blocksProcessed / bytesProcessed); the V1 caller
> passes its deduplicated counts unchanged, since V1 processes each block once.
> A unit test in {{TestBlockDeletingService}} marks two delete transactions for
> the same block, runs one deletion pass, and asserts the persisted pending
> count and bytes both drain to 0. This addresses the root cause of the
> publicly tracked HDDS-8063 and reduces the {{resetPendingDeleteBlockCount}}
> workaround to a safety net.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
