Siyao Meng created HDDS-17573:
---------------------------------
Summary: OM follower that finishes a checkpoint install is never
reloaded by Ratis when appends it accepted earlier reach its log after the
notification
Key: HDDS-17573
URL: https://issues.apache.org/jira/browse/HDDS-17573
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: TestBugMC4InstallNeverAcknowledged.java
h3. Mechanism
A follower OM installs a leader checkpoint when Ratis calls
{{OzoneManagerStateMachine.notifyInstallSnapshotFromLeader}}.
{{OzoneManager.installSnapshotFromLeader}} downloads and installs the
checkpoint asynchronously and completes the future with the checkpoint
TermIndex; it does not check whether the follower's log already reaches the
leader's first index. In Ratis 3.3.1, {{SnapshotInstallationHandler}} keeps
{{inProgressInstallSnapshotIndex}} set while the install runs, and
{{RaftServerImpl.checkInconsistentAppendEntries}} rejects every append
meanwhile ("installation is in progress") with the follower's log next index. A
finished install is used (state machine reload, SNAPSHOT_INSTALLED) only when
the next install notification arrives; that handling comes from RATIS-1481. The
leader sends one only while the follower's next index is below the leader's
first log index ({{LogAppender.shouldInstallSnapshot}}).
An append the follower accepted before the notification can reach its log only
afterwards: with append compose (Ratis 3.2.0 and later) each accepted append is
written after the previous one is flushed, and its consistency check does not
run again. Once it lands, the follower reports a next index above the leader's
first index, the leader stops notifying, and the finished install is never
used. The comment in {{OzoneManager.installSnapshotFromLeader}} ("The follower
refuses appendEntries while this install is in progress, so its commit index is
fixed") does not hold either: the commit index also rises when that append
lands.
h3. Trigger
# Follower F accepts appends from leader L1 while its log flush is slow, so the
accepted entries are not in its log yet. F misses L1's last entries.
# Another OM L2 takes a snapshot and purges its log (default
{{ozone.om.ratis.log.purge.upto.snapshot.index=true}}) past F's log end but not
past what F accepted, and leadership moves to L2 (admin transfer or election).
# L2 sees F below its first log index and notifies F, F starts the install.
# F's flush completes, the accepted entries land, and F's log next index passes
L2's first index.
h3. Impact
* F finishes the install but Ratis never reloads its state machine and F
refuses every append from L2. It stays out of the quorum until it restarts, a
leader purges past its next index again (about 1M transactions with the default
{{ozone.om.ratis.log.purge.gap}}), or a leader whose first log index is above
F's next index takes over (argued). An idle cluster does not recover.
* The cluster stays available with the other two OMs, but has no failure margin
left.
* The trigger is rare: it needs a follower log stall that spans a leadership
change and the new leader's purge, with the purge point falling inside the
stalled range.
* The OM level effect is argued from the code, not run in an OM cluster.
HDDS-16932 reaches the same Ratis state for SCM when leadership moves during
the download; OM is exposed to that trigger as well, and since it needs no slow
disk it is probably the more likely route for OM (argued). The SCM side fix
suggested there (fail the future when the leader changed) does not carry over
to OM, because OM swaps the DB before it completes the future. Here the state
is reached when the leader that notified stays leader, through an append the
follower accepted before the notification.
h3. Reproduction
PASS on unmodified source (the defect is present), 3 of 3 runs, about 22 s
each. [^TestBugMC4InstallNeverAcknowledged.java]
({{hadoop-ozone/ozone-manager}}) runs three in-process Ratis 3.3.1 servers over
gRPC with OM's install settings (notification only, purge up to the snapshot
index), small log segments and purge gap 1. Its state machine installs the
leader's snapshot asynchronously and returns its TermIndex, as OM does. F's log
flush is stalled once through the state machine's {{flush}} callback (standing
in for a slow fsync), L1's last appends to F are held with the Ratis
{{RaftServerImpl.appendEntries}} test hook, L2 takes a snapshot through the
snapshot management API and leadership is transferred to it. Observed in one
run: F accepted up to index 79 with log next index 21; L2's first index after
purge 65; L2 notified F; after the flush was released F replied {{nextIndex=80,
followerCommit=79}} (17 before). Other runs differ by a few indices before the
release; the outcome was the same: F's install returned (t:1, i:80); after 30
more writes L2 had committed 145 while F stayed at applied index 79, with no
reload, no SNAPSHOT_INSTALLED and no further notification.
h3. Suggested fix
The fix belongs in Ratis: use a finished state machine install on the append
path too. When an append or heartbeat arrives and the install has finished,
reload the state machine as the notification path does, clear the in progress
index and reply with the new next index. That also closes HDDS-16932 and the
case where the state machine returns null. A check in OM alone cannot close it:
failing the install future before the DB swap when the follower's log already
contains the leader's first TermIndex is safe but only narrows the window,
because the write can still land afterwards, and after the swap OM cannot fail
the future without applying entries the checkpoint already contains.
As a possible mitigation only, turning append compose off for OM
({{ozone.om.ha.raft.server.log.append-entries.compose.enabled=false}}) keeps
accepted entries in the follower's log at once: with it, the reproduction's
precondition fails (the follower's log next index is right after what it
accepted, so the window does not open). This was tried in two runs of the
reproduction only, is not a fix (it leaves the short window between the check
and the append), its throughput cost is not measured, and it does not cover
HDDS-16932's trigger.
h3. Patch
No patch. The reproduction is attached.
Found by TLA+ model checking and code review of the OM and SCM Ratis state
machine integration (leader readiness, snapshot install and log failure
handling) under HDDS-15926, on commit 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0.
Checked against HDDS and RATIS issues and apache/ozone pull requests for
duplicates before filing. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
