[
https://issues.apache.org/jira/browse/HDDS-16442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16442:
----------------------------------
Labels: pull-request-available (was: )
> After OM upgrade, pre-existing DONE snapshot diff jobs are permanently
> unreadable
> ---------------------------------------------------------------------------------
>
> Key: HDDS-16442
> URL: https://issues.apache.org/jira/browse/HDDS-16442
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Labels: pull-request-available
> Attachments: HDDS-16442.001.patch
>
>
> h3. Finding
> After a routine OM upgrade, every pre-existing DONE snapshot-diff job
> deserializes with a null largestEntryKey, so the first client read of the
> report trips the integrity check and hits an uncaught NPE (wrong-key
> updateJobStatus) instead of the designed retriable error; the report is
> intact yet permanently unreadable and the job is stuck DONE (not
> resubmittable) until the retention TTL GC, affecting all old jobs broadly
> rather than via a rare race.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding CR-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-snapshot-diff-jobs
> Guidance:
> campaigns/ozone-9fbf9ee/targets/009-om-snapshot-diff-jobs/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-snapshot-diff-jobs|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16430 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition (reachable through a routine upgrade): a snapshot diff job for
> (fromSnapshot, toSnapshot) completed and was persisted as DONE by an OM build
> that predates the largestEntryKey field on SnapshotDiffJob. After the OM is
> upgraded, that job deserializes with largestEntryKey == null and stays in the
> active job table (loadJobsOnStartUp leaves DONE jobs untouched), with its
> report rows fully intact.
> 1. Client reads the completed diff: getSnapshotDiffReport routes the DONE job
> to createPageResponse.
> 2. On the last page, checkReportsIntegrity sees largestEntryKey == null and
> treats it as an integrity failure.
> 3. The recovery path calls updateJobStatus(diffJob.getJobId(), DONE, FAILED).
> The job table is keyed by the dedup jobKey (fromSnapshotId + DELIMITER +
> toSnapshotId), not by the per-generation jobId, so the lookup returns null
> and dereferencing its status throws NullPointerException.
> 4. The designed retriable IOException ("Report integrity check failed. Retry
> after: ...") is never thrown, the DONE to FAILED transition never lands, and
> the job is stuck DONE (not a resubmittable status), so the client cannot
> recover until the report-retention TTL GC. Every read repeats the NPE.
> The same NPE occurs for any genuine last-page integrity mismatch (non-null
> largestEntryKey that disagrees with the report tail), since the recovery call
> always passes the wrong key.
> h3. Root cause
> hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
> checkReportsIntegrity called updateJobStatus(diffJob.getJobId(), DONE,
> FAILED), passing the jobId where updateJobStatus expects the dedup jobKey (it
> does snapDiffJobTable.get(jobKey)). Two problems compound: (1) a null
> largestEntryKey (legacy DONE job) is treated as a failure instead of a "no
> stored last entry" value, and (2) updateJobStatus has no null guard, so the
> wrong-key lookup NPEs on getStatus().
> h3. Suggested fix
> Thread the dedup jobKey down through createPageResponse into
> checkReportsIntegrity and pass it to updateJobStatus. Treat a null
> largestEntryKey on a DONE job as a legacy value and skip the last-page
> integrity assertion so upgraded jobs stay readable. Null-guard
> updateJobStatus so a missing row logs and returns instead of throwing.
> Regression tests cover both the legacy-null (stays readable) and
> genuine-mismatch (flips to FAILED via jobKey, throws the retriable
> IOException) cases. See [^HDDS-16442.001.patch].
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
