[
https://issues.apache.org/jira/browse/HDDS-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16441:
------------------------------
Description:
h3. Finding
A cancelled generation's task publishes its result as the new generation's
DONE, so the resubmit path (submitSnapshotDiff, cancelSnapshotDiff,
submitSnapshotDiff) leaves a report the reader cannot read (jobId-prefixed key
mismatch fails the integrity check) and NPEs on read; the job is stuck DONE, is
not client-resubmittable, and the client's legitimately recomputed diff is
discarded, with no recovery until the report-retention TTL GC.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-snapshot-diff-jobs
Guidance:
campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
1. Client submits a diff for (fromSnapshot, toSnapshot); generation gen1 runs
(jobKey status IN_PROGRESS), gen1's executor task is live.
2. Client cancels: cancelSnapshotDiff flips the jobKey IN_PROGRESS to CANCELLED
but does not interrupt the still-running gen1 task.
3. Client resubmits (CANCELLED is resubmittable): a new generation gen2 with a
fresh jobId is created under the same jobKey and the jobKey status is flipped
back to IN_PROGRESS.
4. The orphaned gen1 task reaches its publish step. Its guard checks only that
the jobKey status is IN_PROGRESS (now true again for gen2), so it calls
updateJobStatusToDone and publishes gen1's result as the current generation's
DONE.
5. The stored generation is now gen2's jobId but the report was published for
gen1. The report is unreadable (the report keys embed the jobId, so the
integrity check fails) and the job is stuck DONE (not resubmittable), so the
client's legitimately recomputed gen2 diff is discarded until the
report-retention GC.
h3. Root cause
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
the running task's publish (updateJobStatusToDone) was guarded only by status
== IN_PROGRESS, never by the task's own generation. Cancellation is cooperative
and a resubmit reuses the same jobKey with a new jobId, so a superseded
generation's task could publish onto the new generation.
h3. Suggested fix
Thread the running task's jobId into updateJobStatusToDone and publish only
when the stored SnapshotDiffJob.getJobId() still equals the task's jobId;
otherwise log and skip. This makes publication generation-aware so a cancelled
generation's orphaned task cannot publish onto a newer one. The method is also
null-guarded. A unit test asserts a superseded-generation publish is rejected
while the live generation publishes normally. See [^HDDS-16441.001.patch].
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge. A deterministic end-to-end reproduction of the
race would require placing the orphaned task at its publish step across the
resubmit (an executor rendezvous seam not present in shipping code); the test
therefore pins the generation-match invariant on the publish path directly.
was:
h3. Finding
A cancelled generation's task publishes its result as the new generation's
DONE, so the resubmit path (submitSnapshotDiff, cancelSnapshotDiff,
submitSnapshotDiff) leaves a report the reader cannot read (jobId-prefixed key
mismatch fails the integrity check) and NPEs on read; the job is stuck DONE, is
not client-resubmittable, and the client's legitimately recomputed diff is
discarded, with no recovery until the report-retention TTL GC.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-snapshot-diff-jobs
Guidance:
campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> Cancelled snapshot diff generation publishes an unreadable DONE report on
> resubmit
> ----------------------------------------------------------------------------------
>
> Key: HDDS-16441
> URL: https://issues.apache.org/jira/browse/HDDS-16441
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Critical
> Attachments: HDDS-16441.001.patch
>
>
> h3. Finding
> A cancelled generation's task publishes its result as the new generation's
> DONE, so the resubmit path (submitSnapshotDiff, cancelSnapshotDiff,
> submitSnapshotDiff) leaves a report the reader cannot read (jobId-prefixed
> key mismatch fails the integrity check) and NPEs on read; the job is stuck
> DONE, is not client-resubmittable, and the client's legitimately recomputed
> diff is discarded, with no recovery until the report-retention TTL GC.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-snapshot-diff-jobs
> Guidance:
> campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> 1. Client submits a diff for (fromSnapshot, toSnapshot); generation gen1 runs
> (jobKey status IN_PROGRESS), gen1's executor task is live.
> 2. Client cancels: cancelSnapshotDiff flips the jobKey IN_PROGRESS to
> CANCELLED but does not interrupt the still-running gen1 task.
> 3. Client resubmits (CANCELLED is resubmittable): a new generation gen2 with
> a fresh jobId is created under the same jobKey and the jobKey status is
> flipped back to IN_PROGRESS.
> 4. The orphaned gen1 task reaches its publish step. Its guard checks only
> that the jobKey status is IN_PROGRESS (now true again for gen2), so it calls
> updateJobStatusToDone and publishes gen1's result as the current generation's
> DONE.
> 5. The stored generation is now gen2's jobId but the report was published for
> gen1. The report is unreadable (the report keys embed the jobId, so the
> integrity check fails) and the job is stuck DONE (not resubmittable), so the
> client's legitimately recomputed gen2 diff is discarded until the
> report-retention GC.
> h3. Root cause
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
> the running task's publish (updateJobStatusToDone) was guarded only by
> status == IN_PROGRESS, never by the task's own generation. Cancellation is
> cooperative and a resubmit reuses the same jobKey with a new jobId, so a
> superseded generation's task could publish onto the new generation.
> h3. Suggested fix
> Thread the running task's jobId into updateJobStatusToDone and publish only
> when the stored SnapshotDiffJob.getJobId() still equals the task's jobId;
> otherwise log and skip. This makes publication generation-aware so a
> cancelled generation's orphaned task cannot publish onto a newer one. The
> method is also null-guarded. A unit test asserts a superseded-generation
> publish is rejected while the live generation publishes normally. See
> [^HDDS-16441.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge. A deterministic end-to-end reproduction of the
> race would require placing the orphaned task at its publish step across the
> resubmit (an executor rendezvous seam not present in shipping code); the test
> therefore pins the generation-match invariant on the publish path directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
