[
https://issues.apache.org/jira/browse/HDDS-17573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17573:
------------------------------
Attachment: TestBugMC4InstallNeverAcknowledged.java
> OM follower that finishes a checkpoint install is never reloaded by Ratis
> when appends it accepted earlier reach its log after the notification
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17573
> URL: https://issues.apache.org/jira/browse/HDDS-17573
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: TestBugMC4InstallNeverAcknowledged.java
>
>
> h3. Mechanism
> A follower OM installs a leader checkpoint when Ratis calls
> {{OzoneManagerStateMachine.notifyInstallSnapshotFromLeader}}.
> {{OzoneManager.installSnapshotFromLeader}} downloads and installs the
> checkpoint asynchronously and completes the future with the checkpoint
> TermIndex; it does not check whether the follower's log already reaches the
> leader's first index. In Ratis 3.3.1, {{SnapshotInstallationHandler}} keeps
> {{inProgressInstallSnapshotIndex}} set while the install runs, and
> {{RaftServerImpl.checkInconsistentAppendEntries}} rejects every append
> meanwhile ("installation is in progress") with the follower's log next index.
> A finished install is used (state machine reload, SNAPSHOT_INSTALLED) only
> when the next install notification arrives; that handling comes from
> RATIS-1481. The leader sends one only while the follower's next index is
> below the leader's first log index ({{LogAppender.shouldInstallSnapshot}}).
> An append the follower accepted before the notification can reach its log
> only afterwards: with append compose (Ratis 3.2.0 and later) each accepted
> append is written after the previous one is flushed, and its consistency
> check does not run again. Once it lands, the follower reports a next index
> above the leader's first index, the leader stops notifying, and the finished
> install is never used. The comment in
> {{OzoneManager.installSnapshotFromLeader}} ("The follower refuses
> appendEntries while this install is in progress, so its commit index is
> fixed") does not hold either: the commit index also rises when that append
> lands.
> h3. Trigger
> # Follower F accepts appends from leader L1 while its log flush is slow, so
> the accepted entries are not in its log yet. F misses L1's last entries.
> # Another OM L2 takes a snapshot and purges its log (default
> {{ozone.om.ratis.log.purge.upto.snapshot.index=true}}) past F's log end but
> not past what F accepted, and leadership moves to L2 (admin transfer or
> election).
> # L2 sees F below its first log index and notifies F, F starts the install.
> # F's flush completes, the accepted entries land, and F's log next index
> passes L2's first index.
> h3. Impact
> * F finishes the install but Ratis never reloads its state machine and F
> refuses every append from L2. It stays out of the quorum until it restarts, a
> leader purges past its next index again (about 1M transactions with the
> default {{ozone.om.ratis.log.purge.gap}}), or a leader whose first log index
> is above F's next index takes over (argued). An idle cluster does not recover.
> * The cluster stays available with the other two OMs, but has no failure
> margin left.
> * The trigger is rare: it needs a follower log stall that spans a leadership
> change and the new leader's purge, with the purge point falling inside the
> stalled range.
> * The OM level effect is argued from the code, not run in an OM cluster.
> HDDS-16932 reaches the same Ratis state for SCM when leadership moves during
> the download; OM is exposed to that trigger as well, and since it needs no
> slow disk it is probably the more likely route for OM (argued). The SCM side
> fix suggested there (fail the future when the leader changed) does not carry
> over to OM, because OM swaps the DB before it completes the future. Here the
> state is reached when the leader that notified stays leader, through an
> append the follower accepted before the notification.
> h3. Reproduction
> PASS on unmodified source (the defect is present), 3 of 3 runs, about 22 s
> each. [^TestBugMC4InstallNeverAcknowledged.java]
> ({{hadoop-ozone/ozone-manager}}) runs three in-process Ratis 3.3.1 servers
> over gRPC with OM's install settings (notification only, purge up to the
> snapshot index), small log segments and purge gap 1. Its state machine
> installs the leader's snapshot asynchronously and returns its TermIndex, as
> OM does. F's log flush is stalled once through the state machine's {{flush}}
> callback (standing in for a slow fsync), L1's last appends to F are held with
> the Ratis {{RaftServerImpl.appendEntries}} test hook, L2 takes a snapshot
> through the snapshot management API and leadership is transferred to it.
> Observed in one run: F accepted up to index 79 with log next index 21; L2's
> first index after purge 65; L2 notified F; after the flush was released F
> replied {{nextIndex=80, followerCommit=79}} (17 before). Other runs differ by
> a few indices before the release; the outcome was the same: F's install
> returned (t:1, i:80); after 30 more writes L2 had committed 145 while F
> stayed at applied index 79, with no reload, no SNAPSHOT_INSTALLED and no
> further notification.
> h3. Suggested fix
> The fix belongs in Ratis: use a finished state machine install on the append
> path too. When an append or heartbeat arrives and the install has finished,
> reload the state machine as the notification path does, clear the in progress
> index and reply with the new next index. That also closes HDDS-16932 and the
> case where the state machine returns null. A check in OM alone cannot close
> it: failing the install future before the DB swap when the follower's log
> already contains the leader's first TermIndex is safe but only narrows the
> window, because the write can still land afterwards, and after the swap OM
> cannot fail the future without applying entries the checkpoint already
> contains.
> As a possible mitigation only, turning append compose off for OM
> ({{ozone.om.ha.raft.server.log.append-entries.compose.enabled=false}}) keeps
> accepted entries in the follower's log at once: with it, the reproduction's
> precondition fails (the follower's log next index is right after what it
> accepted, so the window does not open). This was tried in two runs of the
> reproduction only, is not a fix (it leaves the short window between the check
> and the append), its throughput cost is not measured, and it does not cover
> HDDS-16932's trigger.
> h3. Patch
> No patch. The reproduction is attached.
> Found by TLA+ model checking and code review of the OM and SCM Ratis state
> machine integration (leader readiness, snapshot install and log failure
> handling) under HDDS-15926, on commit
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. Checked against HDDS and RATIS
> issues and apache/ozone pull requests for duplicates before filing. Generated
> with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
