Siyao Meng created HDDS-16459:
---------------------------------
Summary: Non atomic container snapshot can mark a healthy
container UNHEALTHY after crash with selective WAL loss
Key: HDDS-16459
URL: https://issues.apache.org/jira/browse/HDDS-16459
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Finding
A non-atomic snapshot of (lastAppliedIndex, container2BCSIDMap) can persist a
BCSID from a later log index than the snapshot tag; on a crash that loses the
unsynced container RocksDB WAL while the fsync'd Ratis snapshot survives,
restart validation marks the healthy container spuriously UNHEALTHY. That
marking is durable across restart and reported to SCM, driving unnecessary
re-replication (bounded, recoverable external effect). ENV_LIMITED: the
end-to-end crash with selective WAL loss is not producible in the in-JVM
harness, and a control demonstrated that an atomic snapshot avoids the marking.
h3. Classification
* Verdict: ENV_LIMITED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: dn-container-state-machine
Guidance:
campaigns/ozone-9fbf9ee/targets/020-dn-container-state-machine/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"dn-container-state-machine|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16433 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
