[
https://issues.apache.org/jira/browse/HDDS-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16463:
----------------------------------
Labels: pull-request-available (was: )
> Recon power loss losing an un synced derived write permanently drops an
> applied update
> --------------------------------------------------------------------------------------
>
> Key: HDDS-16463
> URL: https://issues.apache.org/jira/browse/HDDS-16463
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Hesanda Liyanage
> Priority: Critical
> Labels: pull-request-available
> Attachments: HDDS-16463.001.patch
>
>
> h3. Finding
> A power-loss crash that loses the un-synced derived-store write while the
> Derby cursor survives makes startup reconciliation skip reprocess,
> permanently dropping an applied update so every Recon derived-table query
> (container-key API/UI) serves stale/incomplete data with no automatic
> recovery. Production-reachable via the normal delta-sync path plus a
> differential power-loss within Ozone's fault model. ENV_LIMITED: the
> environment limit is the absence of a crash-consistency harness to truncate
> one RocksDB WAL tail relative to another.
> h3. Classification
> * Verdict: ENV_LIMITED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: recon-om-sync
> Guidance:
> campaigns/ozone-9fbf9ee/targets/023-recon-om-sync/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "recon-om-sync|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> Discovered under HDDS-16434 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # During normal delta sync, ReconTaskControllerImpl.processTasks runs each
> derived task, which writes derived data to the container key RocksDB
> (memtable and WAL, not synced by default), and then calls
> recordRunCompletion() to write the task status cursor to the embedded Derby
> TASK_STATUS table, which commits fsync durable. There is no sync barrier
> between the RocksDB write and the Derby commit.
> # A host power loss occurs in that window. The un synced derived RocksDB WAL
> tail (sequence N) is lost, while the fsync durable Derby cursor row (sequence
> N) survives. The earlier OM snapshot RocksDB write at N survives too, because
> it precedes the derived write and its pages are older and more likely already
> flushed.
> # On restart, OzoneManagerServiceProviderImpl.start() reconciliation compares
> the derived task cursor to the delta cursor. Both are at N, so the strict
> comparison is false and no reprocess is queued.
> # The next delta sync reads the OM snapshot store latest sequence (still N),
> so no delta is re fetched either.
> Observed wrong outcome: the applied update is permanently dropped from the
> derived tables. Recon container key API/UI serve stale, incomplete data with
> no automatic recovery.
> Unmasking condition (this issue is ENV_LIMITED): triggering the live lost
> update requires a real differential power loss that drops the un synced
> derived RocksDB WAL tail while the OM snapshot RocksDB and the Derby cursor
> survive. A JVM unit test or a MiniOzoneCluster cannot inject this: a process
> kill leaves both RocksDB WALs in the OS page cache, so nothing is lost on
> reopen, and there is no in test facility to truncate one RocksDB WAL tail
> relative to another. The attached test therefore asserts the durability
> barrier directly rather than crashing.
> h3. Root cause
> In ReconTaskControllerImpl.processTasks, on task success the derived RocksDB
> write is not synced before recordRunCompletion() commits the task status
> cursor to Derby, so after a crash the durable cursor can be ahead of the
> derived data. The startup reconciliation in
> OzoneManagerServiceProviderImpl.start() only reprocesses a derived task when
> its cursor is strictly behind the delta cursor, so it is blind exactly when
> the cursor is durable at the same sequence as data that was lost. HDDS-13956
> added a flush on clean close only, not a write time sync barrier, so this
> window is open on every batch.
> h3. Suggested fix
> Add a write time durability barrier: on task success, sync the derived data
> RocksDB write ahead log with DBStore.flushLog(true) before advancing and
> committing the task status cursor, so the derived data is at least as durable
> as the cursor after any crash. If the sync fails, leave the cursor unadvanced
> so the batch is reprocessed instead of recording an un durable success. The
> attached patch [^HDDS-16463.001.patch] makes this change (in a small helper
> syncReconDbLog) and adds a unit test asserting that a successful batch syncs
> the derived RocksDB WAL with sync true, and never with sync false, before the
> cursor advances.
> Performance note: the barrier currently syncs once per successful task per
> batch; a reviewer may prefer folding it into a single per batch sync. End to
> end reproduction of the permanent lost update still requires the differential
> power loss described above.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
