[
https://issues.apache.org/jira/browse/HDDS-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115904#comment-18115904
]
Ivan Andika edited comment on HDDS-16435 at 9/16/26 8:14 AM:
-------------------------------------------------------------
The skipping of a leadership check is a known compromise, leadership check for
each follower read would kill the already bad follower read performance. The
split brain can exists, but split brain exists in a rare case where there is a
small window in a network partition where two leaders believe they are the
leaders before one eventually steps down.
was (Author: JIRAUSER298977):
The skipping of a leadership check is a known compromise, leadership check for
each follower read would kill the already bad follower read performance. The
split brain can exists, but split brain exists in a rare case where there is a
small window in a network partition where two leaders believe they are the
leaders.
> OM LINEARIZABLE_ALLOW_FOLLOWER read can return stale data from a superseded
> leader
> ----------------------------------------------------------------------------------
>
> Key: HDDS-16435
> URL: https://issues.apache.org/jira/browse/HDDS-16435
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Ivan Andika
> Priority: Major
> Attachments: HDDS-16435.001.patch
>
>
> h3. Finding
> Under the opt-in leader-skip config (LINEARIZABLE_ALLOW_FOLLOWER), a
> superseded leader still self-reporting LEADER_AND_READY serves a raw local
> read with no ReadIndex or leadership recheck, so a client that explicitly
> asked for no-stale reads can silently receive committed data older than an
> acknowledged write during a partition-induced split-brain window. Three
> related model-checking findings share this root cause: MC-1 (the RPC read
> path), MC-2 (LINEARIZABLE_ALLOW_FOLLOWER routed to a stale-capable raw local
> read while LINEARIZABLE_LEADER_ONLY keeps ReadIndex), and CR-2 (on leader
> fall-back the gRPC/S3 gateway transport re-hints to DEFAULT while the RPC
> transport keeps LINEARIZABLE_ALLOW_FOLLOWER). Bounded, externally observable
> linearizability (safety) violation via a missing leadership/quorum check.
> ENV_LIMITED: MiniOzoneHACluster cannot inject the asymmetric partition, so
> the routing/hint divergence was reproduced but the live stale value was not.
> h3. Classification
> * Verdict: ENV_LIMITED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding
> MC-1/MC-2/CR-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-follower-read
> Guidance:
> campaigns/ozone-9fbf9ee/targets/001-om-follower-read/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-follower-read|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16428 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # Deploy a 3 node OM HA Raft group with linearizable read enabled
> (ozone.om.read.option=LINEARIZABLE) and set
> ozone.om.allow.leader.skip.linearizable.read=true.
> # A client issues a read carrying the LINEARIZABLE_ALLOW_FOLLOWER consistency
> hint. The proto documents this level as ensuring there are no stale reads,
> and when follower read is enabled it is also the client default read
> consistency.
> # The request lands on an OM that still believes it is the leader:
> getLeaderStatus returns LEADER_AND_READY, a value derived purely from the
> local Ratis role without any quorum confirmation.
> # Because the leader skip flag is set,
> submitReadRequestToOmLinearizableAllowFollower serves the request with a raw
> local read (handler.handleReadRequest) and skips the ReadIndex barrier.
> # If that OM has been superseded by a new leader during a partition (its term
> is behind the new leader's term), its local committed state is behind the
> cluster, so the client that explicitly asked for no stale reads receives
> stale data.
> Unmasking condition: the stale value is only observed when the serving OM is
> a superseded leader that still reports LEADER_AND_READY. That requires an
> asymmetric network partition isolating the old leader from the quorum while
> it stays reachable to clients. A current leader returns the fresh value, so
> the routing divergence is masked in a healthy cluster, and the partition
> cannot be injected in the unit or MiniOzoneHACluster test harness. The
> routing defect itself is deterministic; only the live stale read is
> environment limited.
> h3. Root cause
> The read routing in OzoneManagerProtocolServerSideTranslatorPB treats the two
> linearizable levels inconsistently on a ready leader.
> submitReadRequestToOmLinearizableLeaderOnly always uses the ReadIndex path on
> LEADER_AND_READY, but submitReadRequestToOmLinearizableAllowFollower first
> checks isAllowLeaderSkipLinearizableRead and, when it is set, returns a raw
> local read with no ReadIndex confirmation. getLeaderStatus
> (OzoneManagerRatisServer) reflects only the node's local Ratis role, so a
> superseded leader still passes the LEADER_AND_READY guard for a window. As a
> result an explicit LINEARIZABLE_ALLOW_FOLLOWER request is silently downgraded
> to the DEFAULT serve local behavior, defeating the documented no stale reads
> guarantee and diverging from LINEARIZABLE_LEADER_ONLY. The leader skip flag
> is a performance optimization meant for the no hint path (where DEFAULT
> already permits a stale read window during a partition), and its config
> description discloses no consistency impact on the explicit linearizable
> levels.
> A related transport gap: on follower read fall back the RPC failover proxy
> keeps the LINEARIZABLE_ALLOW_FOLLOWER hint while the gRPC transport re hints
> the leader retry to DEFAULT, so the two transports can disagree on the
> effective consistency of the same fall back read. This shares the same
> underlying hazard (an explicit linearizable follower read being downgraded to
> a stale capable leader local read) and is noted here for the reviewer; the
> attached patch addresses the server side leader skip downgrade, which is the
> primary and common path.
> h3. Suggested fix
> Do not apply the leader skip optimization to an explicit
> LINEARIZABLE_ALLOW_FOLLOWER read. The attached patch [^HDDS-16435.001.patch]
> removes the leader skip branch from
> submitReadRequestToOmLinearizableAllowFollower so that on a ready leader the
> request always takes the ReadIndex path, matching LINEARIZABLE_LEADER_ONLY.
> The optimization stays active for the no hint (DEFAULT) read path, whose
> contract already allows a stale read window.
> The added unit test (TestOmReadConsistencyRouting) drives the read routing
> through processRequest with the leader skip flag enabled and asserts that a
> LINEARIZABLE_ALLOW_FOLLOWER read on a LEADER_AND_READY node takes the
> ReadIndex path and never the leader skip local read, and that both
> linearizable levels route identically. Both assertions fail on the current
> code and pass with the fix. A full end to end reproduction of the stale value
> requires a superseded leader window from an asymmetric partition, which the
> in process test harness cannot inject.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
