[
https://issues.apache.org/jira/browse/HDDS-16442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16442:
------------------------------
Description:
h3. Finding
After a routine OM upgrade, every pre-existing DONE snapshot-diff job
deserializes with a null largestEntryKey, so the first client read of the
report trips the integrity check and hits an uncaught NPE (wrong-key
updateJobStatus) instead of the designed retriable error; the report is intact
yet permanently unreadable and the job is stuck DONE (not resubmittable) until
the retention TTL GC, affecting all old jobs broadly rather than via a rare
race.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-snapshot-diff-jobs
Guidance:
campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
Precondition (reachable through a routine upgrade): a snapshot diff job for
(fromSnapshot, toSnapshot) completed and was persisted as DONE by an OM build
that predates the largestEntryKey field on SnapshotDiffJob. After the OM is
upgraded, that job deserializes with largestEntryKey == null and stays in the
active job table (loadJobsOnStartUp leaves DONE jobs untouched), with its
report rows fully intact.
1. Client reads the completed diff: getSnapshotDiffReport routes the DONE job
to createPageResponse.
2. On the last page, checkReportsIntegrity sees largestEntryKey == null and
treats it as an integrity failure.
3. The recovery path calls updateJobStatus(diffJob.getJobId(), DONE, FAILED).
The job table is keyed by the dedup jobKey (fromSnapshotId + DELIMITER +
toSnapshotId), not by the per-generation jobId, so the lookup returns null and
dereferencing its status throws NullPointerException.
4. The designed retriable IOException ("Report integrity check failed. Retry
after: ...") is never thrown, the DONE to FAILED transition never lands, and
the job is stuck DONE (not a resubmittable status), so the client cannot
recover until the report-retention TTL GC. Every read repeats the NPE.
The same NPE occurs for any genuine last-page integrity mismatch (non-null
largestEntryKey that disagrees with the report tail), since the recovery call
always passes the wrong key.
h3. Root cause
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
checkReportsIntegrity called updateJobStatus(diffJob.getJobId(), DONE,
FAILED), passing the jobId where updateJobStatus expects the dedup jobKey (it
does snapDiffJobTable.get(jobKey)). Two problems compound: (1) a null
largestEntryKey (legacy DONE job) is treated as a failure instead of a "no
stored last entry" value, and (2) updateJobStatus has no null guard, so the
wrong-key lookup NPEs on getStatus().
h3. Suggested fix
Thread the dedup jobKey down through createPageResponse into
checkReportsIntegrity and pass it to updateJobStatus. Treat a null
largestEntryKey on a DONE job as a legacy value and skip the last-page
integrity assertion so upgraded jobs stay readable. Null-guard updateJobStatus
so a missing row logs and returns instead of throwing. Regression tests cover
both the legacy-null (stays readable) and genuine-mismatch (flips to FAILED via
jobKey, throws the retriable IOException) cases. See [^HDDS-16442.001.patch].
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
After a routine OM upgrade, every pre-existing DONE snapshot-diff job
deserializes with a null largestEntryKey, so the first client read of the
report trips the integrity check and hits an uncaught NPE (wrong-key
updateJobStatus) instead of the designed retriable error; the report is intact
yet permanently unreadable and the job is stuck DONE (not resubmittable) until
the retention TTL GC, affecting all old jobs broadly rather than via a rare
race.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-snapshot-diff-jobs
Guidance:
campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> After OM upgrade, pre-existing DONE snapshot diff jobs are permanently
> unreadable
> ---------------------------------------------------------------------------------
>
> Key: HDDS-16442
> URL: https://issues.apache.org/jira/browse/HDDS-16442
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-16442.001.patch
>
>
> h3. Finding
> After a routine OM upgrade, every pre-existing DONE snapshot-diff job
> deserializes with a null largestEntryKey, so the first client read of the
> report trips the integrity check and hits an uncaught NPE (wrong-key
> updateJobStatus) instead of the designed retriable error; the report is
> intact yet permanently unreadable and the job is stuck DONE (not
> resubmittable) until the retention TTL GC, affecting all old jobs broadly
> rather than via a rare race.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-snapshot-diff-jobs
> Guidance:
> campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition (reachable through a routine upgrade): a snapshot diff job for
> (fromSnapshot, toSnapshot) completed and was persisted as DONE by an OM build
> that predates the largestEntryKey field on SnapshotDiffJob. After the OM is
> upgraded, that job deserializes with largestEntryKey == null and stays in the
> active job table (loadJobsOnStartUp leaves DONE jobs untouched), with its
> report rows fully intact.
> 1. Client reads the completed diff: getSnapshotDiffReport routes the DONE job
> to createPageResponse.
> 2. On the last page, checkReportsIntegrity sees largestEntryKey == null and
> treats it as an integrity failure.
> 3. The recovery path calls updateJobStatus(diffJob.getJobId(), DONE, FAILED).
> The job table is keyed by the dedup jobKey (fromSnapshotId + DELIMITER +
> toSnapshotId), not by the per-generation jobId, so the lookup returns null
> and dereferencing its status throws NullPointerException.
> 4. The designed retriable IOException ("Report integrity check failed. Retry
> after: ...") is never thrown, the DONE to FAILED transition never lands, and
> the job is stuck DONE (not a resubmittable status), so the client cannot
> recover until the report-retention TTL GC. Every read repeats the NPE.
> The same NPE occurs for any genuine last-page integrity mismatch (non-null
> largestEntryKey that disagrees with the report tail), since the recovery call
> always passes the wrong key.
> h3. Root cause
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
> checkReportsIntegrity called updateJobStatus(diffJob.getJobId(), DONE,
> FAILED), passing the jobId where updateJobStatus expects the dedup jobKey (it
> does snapDiffJobTable.get(jobKey)). Two problems compound: (1) a null
> largestEntryKey (legacy DONE job) is treated as a failure instead of a "no
> stored last entry" value, and (2) updateJobStatus has no null guard, so the
> wrong-key lookup NPEs on getStatus().
> h3. Suggested fix
> Thread the dedup jobKey down through createPageResponse into
> checkReportsIntegrity and pass it to updateJobStatus. Treat a null
> largestEntryKey on a DONE job as a legacy value and skip the last-page
> integrity assertion so upgraded jobs stay readable. Null-guard
> updateJobStatus so a missing row logs and returns instead of throwing.
> Regression tests cover both the legacy-null (stays readable) and
> genuine-mismatch (flips to FAILED via jobKey, throws the retriable
> IOException) cases. See [^HDDS-16442.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
