[
https://issues.apache.org/jira/browse/HDDS-16462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16462:
------------------------------
Description:
h3. Finding
A full-snapshot swap onto an earlier OM lineage plus a crash before the async
reprocess becomes durable leaves derived tables permanently built against the
old OM lineage, because epoch-blind strict-less-than reconciliation judges the
stale task caught up; Recon REST/API/UI then serves rows the current OM lineage
does not contain, with no self-heal. Triggerable via the documented
delta-failure to full-snapshot fallback after an OM DR restore or re-bootstrap.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: recon-om-sync
Guidance: campaigns/ozone-9fbf9ee/targets/023-recon-om-sync/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"recon-om-sync|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16434 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
# An OM is restored or re-bootstrapped (disaster recovery), so it presents a
new lineage whose RocksDB sequence number is lower than or equal to what Recon
has already processed against the previous lineage.
# Recon's next delta fetch (getDBUpdates since the old high sequence) fails
with SequenceNumberNotFoundException (the same path exercised by
testSyncDataFromOMFullSnapshotForSNNFE), so Recon falls back to a full snapshot
install via executeFullSnapshot.
# executeFullSnapshot sets both the OmSnapshotRequest and OmDeltaRequest
cursors to the new (lower) sequence number, persists them to the embedded Derby
TASK_STATUS table (fsync durable), and only then queues an asynchronous derived
table reprocess (queueReInitializationEvent). The queued reprocess is held in
an in-memory buffer and is not durable.
# Recon crashes after the cursors are persisted but before the async reprocess
completes.
# On restart, OzoneManagerServiceProviderImpl.start() runs startup
reconciliation. For each derived task it queues a reprocess only when
derivedTask.getLastUpdatedSeqNumber() is strictly less than
deltaTask.getLastUpdatedSeqNumber(). The stale derived cursor from the old
lineage is numerically greater than the new (lower) delta cursor, so the strict
less than is false and the derived task is judged caught up.
Observed wrong outcome: the derived tables (for example the ContainerKeyMapper
CONTAINER_KEY tables) stay built against the old OM lineage. Recon REST/API/UI
serve container key rows that the current OM lineage does not contain, and
there is no self heal.
h3. Root cause
In OzoneManagerServiceProviderImpl.start(), the startup reconciliation filter
selects a derived task for reprocess only when
taskStatusUpdater.getLastUpdatedSeqNumber().compareTo(deltaTaskStatusUpdater.getLastUpdatedSeqNumber())
is strictly less than zero. That comparison assumes one monotonic OM sequence
space, but a full snapshot swap in executeFullSnapshot can install an OM DB
copy whose sequence number is lower than or equal to what a derived task
already recorded against a previous lineage, so the strict less than reads the
higher stale cursor as caught up. Because the queued reprocess is asynchronous
and not durable, a crash before it completes leaves this reconciliation as the
only recovery net, and the net misfires on the lineage regression.
h3. Suggested fix
Change the reconciliation predicate from strict less than to inequality, so a
derived task is reprocessed whenever its persisted cursor does not equal the
delta cursor. This covers both the original lag case (derived cursor behind the
delta cursor) and the lineage regression case (stale derived cursor ahead of a
lower new delta cursor), while equal cursors (the healthy caught up state)
still skip the rebuild. The filter is extracted into a helper
(getTasksToReprocessOnStartup) so the decision is unit testable. The attached
patch [^HDDS-16462.001.patch] makes this change and adds a unit test asserting
that a task whose cursor is ahead of a lower new lineage delta cursor is
reprocessed, that a task lagging the delta cursor is still reprocessed, and
that equal cursors do not trigger a reprocess (no over triggering).
Residual note: a new lineage carrying exactly the same sequence number that
Recon last recorded cannot be distinguished by sequence number alone; detecting
that case would require a lineage or epoch identity on the persisted cursor,
which is out of scope for this change.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
A full-snapshot swap onto an earlier OM lineage plus a crash before the async
reprocess becomes durable leaves derived tables permanently built against the
old OM lineage, because epoch-blind strict-less-than reconciliation judges the
stale task caught up; Recon REST/API/UI then serves rows the current OM lineage
does not contain, with no self-heal. Triggerable via the documented
delta-failure to full-snapshot fallback after an OM DR restore or re-bootstrap.
h3. Classification
* Verdict: REPRODUCED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding MC-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: recon-om-sync
Guidance: campaigns/ozone-9fbf9ee/targets/023-recon-om-sync/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"recon-om-sync|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16434 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> Recon full snapshot onto an earlier OM lineage plus crash permanently serves
> stale derived data
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-16462
> URL: https://issues.apache.org/jira/browse/HDDS-16462
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-16462.001.patch
>
>
> h3. Finding
> A full-snapshot swap onto an earlier OM lineage plus a crash before the async
> reprocess becomes durable leaves derived tables permanently built against the
> old OM lineage, because epoch-blind strict-less-than reconciliation judges
> the stale task caught up; Recon REST/API/UI then serves rows the current OM
> lineage does not contain, with no self-heal. Triggerable via the documented
> delta-failure to full-snapshot fallback after an OM DR restore or
> re-bootstrap.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: recon-om-sync
> Guidance:
> campaigns/ozone-9fbf9ee/targets/023-recon-om-sync/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "recon-om-sync|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> Discovered under HDDS-16434 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # An OM is restored or re-bootstrapped (disaster recovery), so it presents a
> new lineage whose RocksDB sequence number is lower than or equal to what
> Recon has already processed against the previous lineage.
> # Recon's next delta fetch (getDBUpdates since the old high sequence) fails
> with SequenceNumberNotFoundException (the same path exercised by
> testSyncDataFromOMFullSnapshotForSNNFE), so Recon falls back to a full
> snapshot install via executeFullSnapshot.
> # executeFullSnapshot sets both the OmSnapshotRequest and OmDeltaRequest
> cursors to the new (lower) sequence number, persists them to the embedded
> Derby TASK_STATUS table (fsync durable), and only then queues an asynchronous
> derived table reprocess (queueReInitializationEvent). The queued reprocess is
> held in an in-memory buffer and is not durable.
> # Recon crashes after the cursors are persisted but before the async
> reprocess completes.
> # On restart, OzoneManagerServiceProviderImpl.start() runs startup
> reconciliation. For each derived task it queues a reprocess only when
> derivedTask.getLastUpdatedSeqNumber() is strictly less than
> deltaTask.getLastUpdatedSeqNumber(). The stale derived cursor from the old
> lineage is numerically greater than the new (lower) delta cursor, so the
> strict less than is false and the derived task is judged caught up.
> Observed wrong outcome: the derived tables (for example the
> ContainerKeyMapper CONTAINER_KEY tables) stay built against the old OM
> lineage. Recon REST/API/UI serve container key rows that the current OM
> lineage does not contain, and there is no self heal.
> h3. Root cause
> In OzoneManagerServiceProviderImpl.start(), the startup reconciliation filter
> selects a derived task for reprocess only when
> taskStatusUpdater.getLastUpdatedSeqNumber().compareTo(deltaTaskStatusUpdater.getLastUpdatedSeqNumber())
> is strictly less than zero. That comparison assumes one monotonic OM
> sequence space, but a full snapshot swap in executeFullSnapshot can install
> an OM DB copy whose sequence number is lower than or equal to what a derived
> task already recorded against a previous lineage, so the strict less than
> reads the higher stale cursor as caught up. Because the queued reprocess is
> asynchronous and not durable, a crash before it completes leaves this
> reconciliation as the only recovery net, and the net misfires on the lineage
> regression.
> h3. Suggested fix
> Change the reconciliation predicate from strict less than to inequality, so a
> derived task is reprocessed whenever its persisted cursor does not equal the
> delta cursor. This covers both the original lag case (derived cursor behind
> the delta cursor) and the lineage regression case (stale derived cursor ahead
> of a lower new delta cursor), while equal cursors (the healthy caught up
> state) still skip the rebuild. The filter is extracted into a helper
> (getTasksToReprocessOnStartup) so the decision is unit testable. The attached
> patch [^HDDS-16462.001.patch] makes this change and adds a unit test
> asserting that a task whose cursor is ahead of a lower new lineage delta
> cursor is reprocessed, that a task lagging the delta cursor is still
> reprocessed, and that equal cursors do not trigger a reprocess (no over
> triggering).
> Residual note: a new lineage carrying exactly the same sequence number that
> Recon last recorded cannot be distinguished by sequence number alone;
> detecting that case would require a lineage or epoch identity on the
> persisted cursor, which is out of scope for this change.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
