[
https://issues.apache.org/jira/browse/HDDS-16462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krishna Kumar Asawa reassigned HDDS-16462:
------------------------------------------
Assignee: Arafat Khan
> Recon full snapshot onto an earlier OM lineage plus crash permanently serves
> stale derived data
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-16462
> URL: https://issues.apache.org/jira/browse/HDDS-16462
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Arafat Khan
> Priority: Critical
> Attachments: HDDS-16462.001.patch
>
>
> h3. Finding
> A full-snapshot swap onto an earlier OM lineage plus a crash before the async
> reprocess becomes durable leaves derived tables permanently built against the
> old OM lineage, because epoch-blind strict-less-than reconciliation judges
> the stale task caught up; Recon REST/API/UI then serves rows the current OM
> lineage does not contain, with no self-heal. Triggerable via the documented
> delta-failure to full-snapshot fallback after an OM DR restore or
> re-bootstrap.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: recon-om-sync
> Guidance:
> campaigns/ozone-9fbf9ee/targets/023-recon-om-sync/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "recon-om-sync|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> Discovered under HDDS-16434 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # An OM is restored or re-bootstrapped (disaster recovery), so it presents a
> new lineage whose RocksDB sequence number is lower than or equal to what
> Recon has already processed against the previous lineage.
> # Recon's next delta fetch (getDBUpdates since the old high sequence) fails
> with SequenceNumberNotFoundException (the same path exercised by
> testSyncDataFromOMFullSnapshotForSNNFE), so Recon falls back to a full
> snapshot install via executeFullSnapshot.
> # executeFullSnapshot sets both the OmSnapshotRequest and OmDeltaRequest
> cursors to the new (lower) sequence number, persists them to the embedded
> Derby TASK_STATUS table (fsync durable), and only then queues an asynchronous
> derived table reprocess (queueReInitializationEvent). The queued reprocess is
> held in an in-memory buffer and is not durable.
> # Recon crashes after the cursors are persisted but before the async
> reprocess completes.
> # On restart, OzoneManagerServiceProviderImpl.start() runs startup
> reconciliation. For each derived task it queues a reprocess only when
> derivedTask.getLastUpdatedSeqNumber() is strictly less than
> deltaTask.getLastUpdatedSeqNumber(). The stale derived cursor from the old
> lineage is numerically greater than the new (lower) delta cursor, so the
> strict less than is false and the derived task is judged caught up.
> Observed wrong outcome: the derived tables (for example the
> ContainerKeyMapper CONTAINER_KEY tables) stay built against the old OM
> lineage. Recon REST/API/UI serve container key rows that the current OM
> lineage does not contain, and there is no self heal.
> h3. Root cause
> In OzoneManagerServiceProviderImpl.start(), the startup reconciliation filter
> selects a derived task for reprocess only when
> taskStatusUpdater.getLastUpdatedSeqNumber().compareTo(deltaTaskStatusUpdater.getLastUpdatedSeqNumber())
> is strictly less than zero. That comparison assumes one monotonic OM
> sequence space, but a full snapshot swap in executeFullSnapshot can install
> an OM DB copy whose sequence number is lower than or equal to what a derived
> task already recorded against a previous lineage, so the strict less than
> reads the higher stale cursor as caught up. Because the queued reprocess is
> asynchronous and not durable, a crash before it completes leaves this
> reconciliation as the only recovery net, and the net misfires on the lineage
> regression.
> h3. Suggested fix
> Change the reconciliation predicate from strict less than to inequality, so a
> derived task is reprocessed whenever its persisted cursor does not equal the
> delta cursor. This covers both the original lag case (derived cursor behind
> the delta cursor) and the lineage regression case (stale derived cursor ahead
> of a lower new delta cursor), while equal cursors (the healthy caught up
> state) still skip the rebuild. The filter is extracted into a helper
> (getTasksToReprocessOnStartup) so the decision is unit testable. The attached
> patch [^HDDS-16462.001.patch] makes this change and adds a unit test
> asserting that a task whose cursor is ahead of a lower new lineage delta
> cursor is reprocessed, that a task lagging the delta cursor is still
> reprocessed, and that equal cursors do not trigger a reprocess (no over
> triggering).
> Residual note: a new lineage carrying exactly the same sequence number that
> Recon last recorded cannot be distinguished by sequence number alone;
> detecting that case would require a lineage or epoch identity on the
> persisted cursor, which is out of scope for this change.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
