[
https://issues.apache.org/jira/browse/HDDS-16439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16439:
------------------------------
Description:
h3. Finding
A parent directory emptied during a scan is queued for expiry only in memory; a
crash after the cursor advances past it but before its deletion flushes strands
the emptied, rule-matching directory so storage/quota is not reclaimed and the
scan still marks complete, violating DirExpiryCompleteness. The external effect
is masked by the next-generation rescan that reclaims it; absent that rescan
the directory would remain stranded. Reachable through the FSO lifecycle scan
under a crash between committed child deletions and the parent's own deletion.
h3. Classification
* Verdict: MASKED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-key-lifecycle
Guidance:
campaigns/ozone-9fbf9ee/targets/006-om-key-lifecycle/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-key-lifecycle|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
# Enable a lifecycle policy on an FSO bucket with a nested tree, for example
Root containing dA (holds a key), dB (holds two empty subdirs dB1, dB2), and dC.
# The recursive scan pops siblings in descending name order. dB1 and dB2 are
found empty and their deletions are committed; dB is then re-evaluated, found
empty, and queued for its own deletion in the in-memory list, but not yet
committed.
# The scan proceeds to dA (which sorts below dB), commits dA's key deletions,
and persists the resume cursor lastScannedDir = dA on that key flush.
# The OM crashes (or loses leadership) before the queued deletion of dB is
flushed. The in-memory queue (deletedDirSet / dirList) is lost.
# On resume the scan restarts from the root. canSkipDir compares paths
lexicographically and skips any directory sorting above the cursor. dB sorts
above dA, so dB is skipped and never re-evaluated in that generation; the
emptied dB is left behind even though the scan reports the generation complete.
Masked: the next scan generation begins a fresh full scan (no cursor); it finds
dB genuinely empty in the directory table and reclaims it. The impact is
delayed reclamation of one generation rather than permanent retention, unless a
crash recurs at the same window every generation.
h3. Root cause
The resume cursor (lastScannedDir) is persisted on the key-deletion flush, but
a directory that has been decided empty and queued for its own deletion lives
only in the in-memory dirList / deletedDirSet until a later flush. Between
committing a lower-named directory's key deletions (which advances the
persisted cursor) and committing the queued higher-named parent's deletion, the
cursor certifies the parent as already scanned while its deletion is not yet
durable. A crash in that window, combined with canSkipDir skipping any path
above the cursor and the in-memory queue being lost, strands the emptied parent
for that scan generation.
h3. Suggested fix
The correct fix must ensure the persisted cursor never certifies a directory as
scanned before that directory's own deletion is durable: for example, advance
the persisted lastScannedDir only once the queued directory deletions up to
that point are committed, or persist the pending directory queue as part of the
scan state so it survives resume. This is a non-trivial change to the scan's
cursor and deletion ordering. Candidate minimal variants were evaluated and
rejected: attaching the cursor to the directory flush reads a stale
lastScannedDir (which is updated on every directory pop) and yields an
incorrect cursor, and flushing the queued directories at the mid-scan key-flush
point changes the delete-request sequence that the existing
fault-injector-based resume tests assert exactly. The fix is therefore deferred
to a focused change with its own review.
The attached patch [^HDDS-16439.001.patch] adds a targeted test that pins the
current resume behavior: an emptied, rule-matching directory whose path sorts
above the injected resume cursor is skipped and left un-reclaimed for that scan
generation. It documents the defect and the resume-skip path so a fix that
stops stranding the parent surfaces for review.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
A parent directory emptied during a scan is queued for expiry only in memory; a
crash after the cursor advances past it but before its deletion flushes strands
the emptied, rule-matching directory so storage/quota is not reclaimed and the
scan still marks complete, violating DirExpiryCompleteness. The external effect
is masked by the next-generation rescan that reclaims it; absent that rescan
the directory would remain stranded. Reachable through the FSO lifecycle scan
under a crash between committed child deletions and the parent's own deletion.
h3. Classification
* Verdict: MASKED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-key-lifecycle
Guidance:
campaigns/ozone-9fbf9ee/targets/006-om-key-lifecycle/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-key-lifecycle|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> FSO lifecycle scan can strand an emptied directory after a crash before its
> deletion flushes
> --------------------------------------------------------------------------------------------
>
> Key: HDDS-16439
> URL: https://issues.apache.org/jira/browse/HDDS-16439
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16439.001.patch
>
>
> h3. Finding
> A parent directory emptied during a scan is queued for expiry only in memory;
> a crash after the cursor advances past it but before its deletion flushes
> strands the emptied, rule-matching directory so storage/quota is not
> reclaimed and the scan still marks complete, violating DirExpiryCompleteness.
> The external effect is masked by the next-generation rescan that reclaims it;
> absent that rescan the directory would remain stranded. Reachable through the
> FSO lifecycle scan under a crash between committed child deletions and the
> parent's own deletion.
> h3. Classification
> * Verdict: MASKED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-key-lifecycle
> Guidance:
> campaigns/ozone-9fbf9ee/targets/006-om-key-lifecycle/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-key-lifecycle|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16429 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # Enable a lifecycle policy on an FSO bucket with a nested tree, for example
> Root containing dA (holds a key), dB (holds two empty subdirs dB1, dB2), and
> dC.
> # The recursive scan pops siblings in descending name order. dB1 and dB2 are
> found empty and their deletions are committed; dB is then re-evaluated, found
> empty, and queued for its own deletion in the in-memory list, but not yet
> committed.
> # The scan proceeds to dA (which sorts below dB), commits dA's key deletions,
> and persists the resume cursor lastScannedDir = dA on that key flush.
> # The OM crashes (or loses leadership) before the queued deletion of dB is
> flushed. The in-memory queue (deletedDirSet / dirList) is lost.
> # On resume the scan restarts from the root. canSkipDir compares paths
> lexicographically and skips any directory sorting above the cursor. dB sorts
> above dA, so dB is skipped and never re-evaluated in that generation; the
> emptied dB is left behind even though the scan reports the generation
> complete.
> Masked: the next scan generation begins a fresh full scan (no cursor); it
> finds dB genuinely empty in the directory table and reclaims it. The impact
> is delayed reclamation of one generation rather than permanent retention,
> unless a crash recurs at the same window every generation.
> h3. Root cause
> The resume cursor (lastScannedDir) is persisted on the key-deletion flush,
> but a directory that has been decided empty and queued for its own deletion
> lives only in the in-memory dirList / deletedDirSet until a later flush.
> Between committing a lower-named directory's key deletions (which advances
> the persisted cursor) and committing the queued higher-named parent's
> deletion, the cursor certifies the parent as already scanned while its
> deletion is not yet durable. A crash in that window, combined with canSkipDir
> skipping any path above the cursor and the in-memory queue being lost,
> strands the emptied parent for that scan generation.
> h3. Suggested fix
> The correct fix must ensure the persisted cursor never certifies a directory
> as scanned before that directory's own deletion is durable: for example,
> advance the persisted lastScannedDir only once the queued directory deletions
> up to that point are committed, or persist the pending directory queue as
> part of the scan state so it survives resume. This is a non-trivial change to
> the scan's cursor and deletion ordering. Candidate minimal variants were
> evaluated and rejected: attaching the cursor to the directory flush reads a
> stale lastScannedDir (which is updated on every directory pop) and yields an
> incorrect cursor, and flushing the queued directories at the mid-scan
> key-flush point changes the delete-request sequence that the existing
> fault-injector-based resume tests assert exactly. The fix is therefore
> deferred to a focused change with its own review.
> The attached patch [^HDDS-16439.001.patch] adds a targeted test that pins the
> current resume behavior: an emptied, rule-matching directory whose path sorts
> above the injected resume cursor is skipped and left un-reclaimed for that
> scan generation. It documents the defect and the resume-skip path so a fix
> that stops stranding the parent surfaces for review.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
