[
https://issues.apache.org/jira/browse/HDDS-16443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16443:
------------------------------
Description:
h3. Finding
On resubmit of a terminal job the cleanup service can purge the freshly-created
live generation between the two persisted writes, deleting the active row and
crashing the client's resubmit RPC with an uncaught NPE; harm is externally
observable but bounded and client-retryable (retry is treated as a fresh
submit), and the window is very small versus the default cleanup interval.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-snapshot-diff-jobs
Guidance:
campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
1. Client submits a diff for (fromSnapshot, toSnapshot); it runs and then
reaches a terminal state (CANCELLED via cancelSnapshotDiff, or FAILED/REJECTED).
2. Client resubmits the same diff. submitSnapshotDiff's resubmit branch calls
createSnapDiffJob, which persists a new generation (fresh jobId) carrying the
previous terminal status, then, in a separate write, calls updateJobStatus to
move it to IN_PROGRESS.
3. SnapshotDiffCleanupService runs on its own thread, takes no
SnapshotDiffManager lock, and purges any row whose status is
FAILED/REJECTED/CANCELLED regardless of age. If it fires in the window between
the two writes, it enqueues the new generation's jobId for purge and deletes
the active row.
4. The submitting thread's updateJobStatus then finds no row and NPEs
(dereferencing a null job); the catch handler updateJobStatusToFailed NPEs the
same way. The resubmitted job is lost and the RPC crashes.
The window is small versus the default cleanup interval, so this is a
low-probability but externally observable failure.
h3. Root cause
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
on resubmit, createSnapDiffJob was called with the previous terminal status,
so the freshly created generation was transiently persisted in an immediately
purge-eligible state. The two persisted writes (create with terminal status,
then transition to IN_PROGRESS) are not atomic with respect to the lock-free
cleanup service, and updateJobStatus/updateJobStatusToFailed had no null-row
guard.
h3. Suggested fix
Create the resubmitted generation as QUEUED, mirroring the fresh-submit path,
so the row is never in an immediately purge-eligible state before it
transitions to IN_PROGRESS (the transition then uses QUEUED as the expected
prior status). Null-guard updateJobStatus and updateJobStatusToFailed so a row
purged out from under them logs and returns instead of throwing. The added unit
test verifies, via a spy, that the resubmit path creates the generation with
QUEUED and never with a terminal status. See [^HDDS-16443.001.patch].
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge. A deterministic end-to-end reproduction of the
race would require interposing the cleanup service between the two persisted
writes, which shipping code does not expose; the test therefore pins the
invariant that the resubmit generation is never created in a purge-eligible
state.
was:
h3. Finding
On resubmit of a terminal job the cleanup service can purge the freshly-created
live generation between the two persisted writes, deleting the active row and
crashing the client's resubmit RPC with an uncaught NPE; harm is externally
observable but bounded and client-retryable (retry is treated as a fresh
submit), and the window is very small versus the default cleanup interval.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-snapshot-diff-jobs
Guidance:
campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> Snapshot diff cleanup can purge a freshly created generation on resubmit and
> crash the client RPC
> -------------------------------------------------------------------------------------------------
>
> Key: HDDS-16443
> URL: https://issues.apache.org/jira/browse/HDDS-16443
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16443.001.patch
>
>
> h3. Finding
> On resubmit of a terminal job the cleanup service can purge the
> freshly-created live generation between the two persisted writes, deleting
> the active row and crashing the client's resubmit RPC with an uncaught NPE;
> harm is externally observable but bounded and client-retryable (retry is
> treated as a fresh submit), and the window is very small versus the default
> cleanup interval.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-snapshot-diff-jobs
> Guidance:
> campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. Client submits a diff for (fromSnapshot, toSnapshot); it runs and then
> reaches a terminal state (CANCELLED via cancelSnapshotDiff, or
> FAILED/REJECTED).
> 2. Client resubmits the same diff. submitSnapshotDiff's resubmit branch calls
> createSnapDiffJob, which persists a new generation (fresh jobId) carrying the
> previous terminal status, then, in a separate write, calls updateJobStatus to
> move it to IN_PROGRESS.
> 3. SnapshotDiffCleanupService runs on its own thread, takes no
> SnapshotDiffManager lock, and purges any row whose status is
> FAILED/REJECTED/CANCELLED regardless of age. If it fires in the window
> between the two writes, it enqueues the new generation's jobId for purge and
> deletes the active row.
> 4. The submitting thread's updateJobStatus then finds no row and NPEs
> (dereferencing a null job); the catch handler updateJobStatusToFailed NPEs
> the same way. The resubmitted job is lost and the RPC crashes.
> The window is small versus the default cleanup interval, so this is a
> low-probability but externally observable failure.
> h3. Root cause
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
> on resubmit, createSnapDiffJob was called with the previous terminal status,
> so the freshly created generation was transiently persisted in an immediately
> purge-eligible state. The two persisted writes (create with terminal status,
> then transition to IN_PROGRESS) are not atomic with respect to the lock-free
> cleanup service, and updateJobStatus/updateJobStatusToFailed had no null-row
> guard.
> h3. Suggested fix
> Create the resubmitted generation as QUEUED, mirroring the fresh-submit path,
> so the row is never in an immediately purge-eligible state before it
> transitions to IN_PROGRESS (the transition then uses QUEUED as the expected
> prior status). Null-guard updateJobStatus and updateJobStatusToFailed so a
> row purged out from under them logs and returns instead of throwing. The
> added unit test verifies, via a spy, that the resubmit path creates the
> generation with QUEUED and never with a terminal status. See
> [^HDDS-16443.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge. A deterministic end-to-end reproduction of the
> race would require interposing the cleanup service between the two persisted
> writes, which shipping code does not expose; the test therefore pins the
> invariant that the resubmit generation is never created in a purge-eligible
> state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
