[
https://issues.apache.org/jira/browse/HDDS-16463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16463:
------------------------------
Description:
h3. Finding
A power-loss crash that loses the un-synced derived-store write while the Derby
cursor survives makes startup reconciliation skip reprocess, permanently
dropping an applied update so every Recon derived-table query (container-key
API/UI) serves stale/incomplete data with no automatic recovery.
Production-reachable via the normal delta-sync path plus a differential
power-loss within Ozone's fault model. ENV_LIMITED: the environment limit is
the absence of a crash-consistency harness to truncate one RocksDB WAL tail
relative to another.
h3. Classification
* Verdict: ENV_LIMITED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: recon-om-sync
Guidance: campaigns/ozone-9fbf9ee/targets/023-recon-om-sync/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"recon-om-sync|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16434 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
# During normal delta sync, ReconTaskControllerImpl.processTasks runs each
derived task, which writes derived data to the container key RocksDB (memtable
and WAL, not synced by default), and then calls recordRunCompletion() to write
the task status cursor to the embedded Derby TASK_STATUS table, which commits
fsync durable. There is no sync barrier between the RocksDB write and the Derby
commit.
# A host power loss occurs in that window. The un synced derived RocksDB WAL
tail (sequence N) is lost, while the fsync durable Derby cursor row (sequence
N) survives. The earlier OM snapshot RocksDB write at N survives too, because
it precedes the derived write and its pages are older and more likely already
flushed.
# On restart, OzoneManagerServiceProviderImpl.start() reconciliation compares
the derived task cursor to the delta cursor. Both are at N, so the strict
comparison is false and no reprocess is queued.
# The next delta sync reads the OM snapshot store latest sequence (still N), so
no delta is re fetched either.
Observed wrong outcome: the applied update is permanently dropped from the
derived tables. Recon container key API/UI serve stale, incomplete data with no
automatic recovery.
Unmasking condition (this issue is ENV_LIMITED): triggering the live lost
update requires a real differential power loss that drops the un synced derived
RocksDB WAL tail while the OM snapshot RocksDB and the Derby cursor survive. A
JVM unit test or a MiniOzoneCluster cannot inject this: a process kill leaves
both RocksDB WALs in the OS page cache, so nothing is lost on reopen, and there
is no in test facility to truncate one RocksDB WAL tail relative to another.
The attached test therefore asserts the durability barrier directly rather than
crashing.
h3. Root cause
In ReconTaskControllerImpl.processTasks, on task success the derived RocksDB
write is not synced before recordRunCompletion() commits the task status cursor
to Derby, so after a crash the durable cursor can be ahead of the derived data.
The startup reconciliation in OzoneManagerServiceProviderImpl.start() only
reprocesses a derived task when its cursor is strictly behind the delta cursor,
so it is blind exactly when the cursor is durable at the same sequence as data
that was lost. HDDS-13956 added a flush on clean close only, not a write time
sync barrier, so this window is open on every batch.
h3. Suggested fix
Add a write time durability barrier: on task success, sync the derived data
RocksDB write ahead log with DBStore.flushLog(true) before advancing and
committing the task status cursor, so the derived data is at least as durable
as the cursor after any crash. If the sync fails, leave the cursor unadvanced
so the batch is reprocessed instead of recording an un durable success. The
attached patch [^HDDS-16463.001.patch] makes this change (in a small helper
syncReconDbLog) and adds a unit test asserting that a successful batch syncs
the derived RocksDB WAL with sync true, and never with sync false, before the
cursor advances.
Performance note: the barrier currently syncs once per successful task per
batch; a reviewer may prefer folding it into a single per batch sync. End to
end reproduction of the permanent lost update still requires the differential
power loss described above.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
A power-loss crash that loses the un-synced derived-store write while the Derby
cursor survives makes startup reconciliation skip reprocess, permanently
dropping an applied update so every Recon derived-table query (container-key
API/UI) serves stale/incomplete data with no automatic recovery.
Production-reachable via the normal delta-sync path plus a differential
power-loss within Ozone's fault model. ENV_LIMITED: the environment limit is
the absence of a crash-consistency harness to truncate one RocksDB WAL tail
relative to another.
h3. Classification
* Verdict: ENV_LIMITED
* Severity: Critical
* Source: Specula TLA+ model checking and confirmation debate, finding MC-1
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: recon-om-sync
Guidance: campaigns/ozone-9fbf9ee/targets/023-recon-om-sync/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"recon-om-sync|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16434 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> Recon power loss losing an un synced derived write permanently drops an
> applied update
> --------------------------------------------------------------------------------------
>
> Key: HDDS-16463
> URL: https://issues.apache.org/jira/browse/HDDS-16463
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments: HDDS-16463.001.patch
>
>
> h3. Finding
> A power-loss crash that loses the un-synced derived-store write while the
> Derby cursor survives makes startup reconciliation skip reprocess,
> permanently dropping an applied update so every Recon derived-table query
> (container-key API/UI) serves stale/incomplete data with no automatic
> recovery. Production-reachable via the normal delta-sync path plus a
> differential power-loss within Ozone's fault model. ENV_LIMITED: the
> environment limit is the absence of a crash-consistency harness to truncate
> one RocksDB WAL tail relative to another.
> h3. Classification
> * Verdict: ENV_LIMITED
> * Severity: Critical
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-1
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: recon-om-sync
> Guidance:
> campaigns/ozone-9fbf9ee/targets/023-recon-om-sync/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "recon-om-sync|apache/ozone|Java|Use the target-specific .prompt-extra.md"
> {code}
> Discovered under HDDS-16434 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # During normal delta sync, ReconTaskControllerImpl.processTasks runs each
> derived task, which writes derived data to the container key RocksDB
> (memtable and WAL, not synced by default), and then calls
> recordRunCompletion() to write the task status cursor to the embedded Derby
> TASK_STATUS table, which commits fsync durable. There is no sync barrier
> between the RocksDB write and the Derby commit.
> # A host power loss occurs in that window. The un synced derived RocksDB WAL
> tail (sequence N) is lost, while the fsync durable Derby cursor row (sequence
> N) survives. The earlier OM snapshot RocksDB write at N survives too, because
> it precedes the derived write and its pages are older and more likely already
> flushed.
> # On restart, OzoneManagerServiceProviderImpl.start() reconciliation compares
> the derived task cursor to the delta cursor. Both are at N, so the strict
> comparison is false and no reprocess is queued.
> # The next delta sync reads the OM snapshot store latest sequence (still N),
> so no delta is re fetched either.
> Observed wrong outcome: the applied update is permanently dropped from the
> derived tables. Recon container key API/UI serve stale, incomplete data with
> no automatic recovery.
> Unmasking condition (this issue is ENV_LIMITED): triggering the live lost
> update requires a real differential power loss that drops the un synced
> derived RocksDB WAL tail while the OM snapshot RocksDB and the Derby cursor
> survive. A JVM unit test or a MiniOzoneCluster cannot inject this: a process
> kill leaves both RocksDB WALs in the OS page cache, so nothing is lost on
> reopen, and there is no in test facility to truncate one RocksDB WAL tail
> relative to another. The attached test therefore asserts the durability
> barrier directly rather than crashing.
> h3. Root cause
> In ReconTaskControllerImpl.processTasks, on task success the derived RocksDB
> write is not synced before recordRunCompletion() commits the task status
> cursor to Derby, so after a crash the durable cursor can be ahead of the
> derived data. The startup reconciliation in
> OzoneManagerServiceProviderImpl.start() only reprocesses a derived task when
> its cursor is strictly behind the delta cursor, so it is blind exactly when
> the cursor is durable at the same sequence as data that was lost. HDDS-13956
> added a flush on clean close only, not a write time sync barrier, so this
> window is open on every batch.
> h3. Suggested fix
> Add a write time durability barrier: on task success, sync the derived data
> RocksDB write ahead log with DBStore.flushLog(true) before advancing and
> committing the task status cursor, so the derived data is at least as durable
> as the cursor after any crash. If the sync fails, leave the cursor unadvanced
> so the batch is reprocessed instead of recording an un durable success. The
> attached patch [^HDDS-16463.001.patch] makes this change (in a small helper
> syncReconDbLog) and adds a unit test asserting that a successful batch syncs
> the derived RocksDB WAL with sync true, and never with sync false, before the
> cursor advances.
> Performance note: the barrier currently syncs once per successful task per
> batch; a reviewer may prefer folding it into a single per batch sync. End to
> end reproduction of the permanent lost update still requires the differential
> power loss described above.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
