[
https://issues.apache.org/jira/browse/HDDS-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16435:
------------------------------
Description:
h3. Finding
Under the opt-in leader-skip config (LINEARIZABLE_ALLOW_FOLLOWER), a superseded
leader still self-reporting LEADER_AND_READY serves a raw local read with no
ReadIndex or leadership recheck, so a client that explicitly asked for no-stale
reads can silently receive committed data older than an acknowledged write
during a partition-induced split-brain window. Three related model-checking
findings share this root cause: MC-1 (the RPC read path), MC-2
(LINEARIZABLE_ALLOW_FOLLOWER routed to a stale-capable raw local read while
LINEARIZABLE_LEADER_ONLY keeps ReadIndex), and CR-2 (on leader fall-back the
gRPC/S3 gateway transport re-hints to DEFAULT while the RPC transport keeps
LINEARIZABLE_ALLOW_FOLLOWER). Bounded, externally observable linearizability
(safety) violation via a missing leadership/quorum check. ENV_LIMITED:
MiniOzoneHACluster cannot inject the asymmetric partition, so the routing/hint
divergence was reproduced but the live stale value was not.
h3. Classification
* Verdict: ENV_LIMITED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding
MC-1/MC-2/CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-follower-read
Guidance:
campaigns/ozone-9fbf9ee/targets/001-om-follower-read/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-follower-read|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16428 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
# Deploy a 3 node OM HA Raft group with linearizable read enabled
(ozone.om.read.option=LINEARIZABLE) and set
ozone.om.allow.leader.skip.linearizable.read=true.
# A client issues a read carrying the LINEARIZABLE_ALLOW_FOLLOWER consistency
hint. The proto documents this level as ensuring there are no stale reads, and
when follower read is enabled it is also the client default read consistency.
# The request lands on an OM that still believes it is the leader:
getLeaderStatus returns LEADER_AND_READY, a value derived purely from the local
Ratis role without any quorum confirmation.
# Because the leader skip flag is set,
submitReadRequestToOmLinearizableAllowFollower serves the request with a raw
local read (handler.handleReadRequest) and skips the ReadIndex barrier.
# If that OM has been superseded by a new leader during a partition (its term
is behind the new leader's term), its local committed state is behind the
cluster, so the client that explicitly asked for no stale reads receives stale
data.
Unmasking condition: the stale value is only observed when the serving OM is a
superseded leader that still reports LEADER_AND_READY. That requires an
asymmetric network partition isolating the old leader from the quorum while it
stays reachable to clients. A current leader returns the fresh value, so the
routing divergence is masked in a healthy cluster, and the partition cannot be
injected in the unit or MiniOzoneHACluster test harness. The routing defect
itself is deterministic; only the live stale read is environment limited.
h3. Root cause
The read routing in OzoneManagerProtocolServerSideTranslatorPB treats the two
linearizable levels inconsistently on a ready leader.
submitReadRequestToOmLinearizableLeaderOnly always uses the ReadIndex path on
LEADER_AND_READY, but submitReadRequestToOmLinearizableAllowFollower first
checks isAllowLeaderSkipLinearizableRead and, when it is set, returns a raw
local read with no ReadIndex confirmation. getLeaderStatus
(OzoneManagerRatisServer) reflects only the node's local Ratis role, so a
superseded leader still passes the LEADER_AND_READY guard for a window. As a
result an explicit LINEARIZABLE_ALLOW_FOLLOWER request is silently downgraded
to the DEFAULT serve local behavior, defeating the documented no stale reads
guarantee and diverging from LINEARIZABLE_LEADER_ONLY. The leader skip flag is
a performance optimization meant for the no hint path (where DEFAULT already
permits a stale read window during a partition), and its config description
discloses no consistency impact on the explicit linearizable levels.
A related transport gap: on follower read fall back the RPC failover proxy
keeps the LINEARIZABLE_ALLOW_FOLLOWER hint while the gRPC transport re hints
the leader retry to DEFAULT, so the two transports can disagree on the
effective consistency of the same fall back read. This shares the same
underlying hazard (an explicit linearizable follower read being downgraded to a
stale capable leader local read) and is noted here for the reviewer; the
attached patch addresses the server side leader skip downgrade, which is the
primary and common path.
h3. Suggested fix
Do not apply the leader skip optimization to an explicit
LINEARIZABLE_ALLOW_FOLLOWER read. The attached patch [^HDDS-16435.001.patch]
removes the leader skip branch from
submitReadRequestToOmLinearizableAllowFollower so that on a ready leader the
request always takes the ReadIndex path, matching LINEARIZABLE_LEADER_ONLY. The
optimization stays active for the no hint (DEFAULT) read path, whose contract
already allows a stale read window.
The added unit test (TestOmReadConsistencyRouting) drives the read routing
through processRequest with the leader skip flag enabled and asserts that a
LINEARIZABLE_ALLOW_FOLLOWER read on a LEADER_AND_READY node takes the ReadIndex
path and never the leader skip local read, and that both linearizable levels
route identically. Both assertions fail on the current code and pass with the
fix. A full end to end reproduction of the stale value requires a superseded
leader window from an asymmetric partition, which the in process test harness
cannot inject.
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
Under the opt-in leader-skip config (LINEARIZABLE_ALLOW_FOLLOWER), a superseded
leader still self-reporting LEADER_AND_READY serves a raw local read with no
ReadIndex or leadership recheck, so a client that explicitly asked for no-stale
reads can silently receive committed data older than an acknowledged write
during a partition-induced split-brain window. Three related model-checking
findings share this root cause: MC-1 (the RPC read path), MC-2
(LINEARIZABLE_ALLOW_FOLLOWER routed to a stale-capable raw local read while
LINEARIZABLE_LEADER_ONLY keeps ReadIndex), and CR-2 (on leader fall-back the
gRPC/S3 gateway transport re-hints to DEFAULT while the RPC transport keeps
LINEARIZABLE_ALLOW_FOLLOWER). Bounded, externally observable linearizability
(safety) violation via a missing leadership/quorum check. ENV_LIMITED:
MiniOzoneHACluster cannot inject the asymmetric partition, so the routing/hint
divergence was reproduced but the live stale value was not.
h3. Classification
* Verdict: ENV_LIMITED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding
MC-1/MC-2/CR-2
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: om-follower-read
Guidance:
campaigns/ozone-9fbf9ee/targets/001-om-follower-read/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=high --keep-original --max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"om-follower-read|apache/ozone|Java|Use the target-specific .prompt-extra.md"
{code}
Discovered under HDDS-16428 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> OM LINEARIZABLE_ALLOW_FOLLOWER read can return stale data from a superseded
> leader
> ----------------------------------------------------------------------------------
>
> Key: HDDS-16435
> URL: https://issues.apache.org/jira/browse/HDDS-16435
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16435.001.patch
>
>
> h3. Finding
> Under the opt-in leader-skip config (LINEARIZABLE_ALLOW_FOLLOWER), a
> superseded leader still self-reporting LEADER_AND_READY serves a raw local
> read with no ReadIndex or leadership recheck, so a client that explicitly
> asked for no-stale reads can silently receive committed data older than an
> acknowledged write during a partition-induced split-brain window. Three
> related model-checking findings share this root cause: MC-1 (the RPC read
> path), MC-2 (LINEARIZABLE_ALLOW_FOLLOWER routed to a stale-capable raw local
> read while LINEARIZABLE_LEADER_ONLY keeps ReadIndex), and CR-2 (on leader
> fall-back the gRPC/S3 gateway transport re-hints to DEFAULT while the RPC
> transport keeps LINEARIZABLE_ALLOW_FOLLOWER). Bounded, externally observable
> linearizability (safety) violation via a missing leadership/quorum check.
> ENV_LIMITED: MiniOzoneHACluster cannot inject the asymmetric partition, so
> the routing/hint divergence was reproduced but the live stale value was not.
> h3. Classification
> * Verdict: ENV_LIMITED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding
> MC-1/MC-2/CR-2
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: om-follower-read
> Guidance:
> campaigns/ozone-9fbf9ee/targets/001-om-follower-read/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=high --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "om-follower-read|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16428 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> # Deploy a 3 node OM HA Raft group with linearizable read enabled
> (ozone.om.read.option=LINEARIZABLE) and set
> ozone.om.allow.leader.skip.linearizable.read=true.
> # A client issues a read carrying the LINEARIZABLE_ALLOW_FOLLOWER consistency
> hint. The proto documents this level as ensuring there are no stale reads,
> and when follower read is enabled it is also the client default read
> consistency.
> # The request lands on an OM that still believes it is the leader:
> getLeaderStatus returns LEADER_AND_READY, a value derived purely from the
> local Ratis role without any quorum confirmation.
> # Because the leader skip flag is set,
> submitReadRequestToOmLinearizableAllowFollower serves the request with a raw
> local read (handler.handleReadRequest) and skips the ReadIndex barrier.
> # If that OM has been superseded by a new leader during a partition (its term
> is behind the new leader's term), its local committed state is behind the
> cluster, so the client that explicitly asked for no stale reads receives
> stale data.
> Unmasking condition: the stale value is only observed when the serving OM is
> a superseded leader that still reports LEADER_AND_READY. That requires an
> asymmetric network partition isolating the old leader from the quorum while
> it stays reachable to clients. A current leader returns the fresh value, so
> the routing divergence is masked in a healthy cluster, and the partition
> cannot be injected in the unit or MiniOzoneHACluster test harness. The
> routing defect itself is deterministic; only the live stale read is
> environment limited.
> h3. Root cause
> The read routing in OzoneManagerProtocolServerSideTranslatorPB treats the two
> linearizable levels inconsistently on a ready leader.
> submitReadRequestToOmLinearizableLeaderOnly always uses the ReadIndex path on
> LEADER_AND_READY, but submitReadRequestToOmLinearizableAllowFollower first
> checks isAllowLeaderSkipLinearizableRead and, when it is set, returns a raw
> local read with no ReadIndex confirmation. getLeaderStatus
> (OzoneManagerRatisServer) reflects only the node's local Ratis role, so a
> superseded leader still passes the LEADER_AND_READY guard for a window. As a
> result an explicit LINEARIZABLE_ALLOW_FOLLOWER request is silently downgraded
> to the DEFAULT serve local behavior, defeating the documented no stale reads
> guarantee and diverging from LINEARIZABLE_LEADER_ONLY. The leader skip flag
> is a performance optimization meant for the no hint path (where DEFAULT
> already permits a stale read window during a partition), and its config
> description discloses no consistency impact on the explicit linearizable
> levels.
> A related transport gap: on follower read fall back the RPC failover proxy
> keeps the LINEARIZABLE_ALLOW_FOLLOWER hint while the gRPC transport re hints
> the leader retry to DEFAULT, so the two transports can disagree on the
> effective consistency of the same fall back read. This shares the same
> underlying hazard (an explicit linearizable follower read being downgraded to
> a stale capable leader local read) and is noted here for the reviewer; the
> attached patch addresses the server side leader skip downgrade, which is the
> primary and common path.
> h3. Suggested fix
> Do not apply the leader skip optimization to an explicit
> LINEARIZABLE_ALLOW_FOLLOWER read. The attached patch [^HDDS-16435.001.patch]
> removes the leader skip branch from
> submitReadRequestToOmLinearizableAllowFollower so that on a ready leader the
> request always takes the ReadIndex path, matching LINEARIZABLE_LEADER_ONLY.
> The optimization stays active for the no hint (DEFAULT) read path, whose
> contract already allows a stale read window.
> The added unit test (TestOmReadConsistencyRouting) drives the read routing
> through processRequest with the leader skip flag enabled and asserts that a
> LINEARIZABLE_ALLOW_FOLLOWER read on a LEADER_AND_READY node takes the
> ReadIndex path and never the leader skip local read, and that both
> linearizable levels route identically. Both assertions fail on the current
> code and pass with the fix. A full end to end reproduction of the stale value
> requires a superseded leader window from an asymmetric partition, which the
> in process test harness cannot inject.
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
