[
https://issues.apache.org/jira/browse/HDDS-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17576:
------------------------------
Attachment: MC-3-fail-install-snapshot-future-for-unknown-scm-leader.patch
> Follower SCM stops applying the Raft log until restart after an install
> snapshot notification from a leader missing from its peer list
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17576
> URL: https://issues.apache.org/jira/browse/HDDS-17576
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.2.0, 1.3.0, 1.2.1, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0,
> 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-3-fail-install-snapshot-future-for-unknown-scm-leader.patch,
> TestBugMC3FollowerStuckAfterInstallNotification.java
>
>
> h3. Mechanism
> {{SCMStateMachine.notifyInstallSnapshotFromLeader}} looks the leader up by
> its Ratis address in {{SCMHANodeDetails.getPeerNodeDetails}} and runs
> {{Preconditions.checkState(leaderDetails.isPresent())}} on the calling Ratis
> thread. When no peer matches it throws {{IllegalStateException}} instead of
> returning a failed future, as the "missing leader info" branch just above it
> does.
> Ratis ({{SnapshotInstallationHandler.notifyStateMachineToInstallSnapshot}},
> 3.3.1) has already set its in-progress install index at that point and clears
> it on failure only in the {{whenComplete}} of the returned future. The
> synchronous exception skips that, and nothing else resets the index. From
> then on the division refuses every {{appendEntries}} ("installation is in
> progress") and answers every later install notification with {{IN_PROGRESS}}
> (or an error, when a later leader's first available index is below the stale
> one) without calling the state machine again, also when the leader is an SCM
> it knows. The index is only in memory, so a restart of the SCM should clear
> it (argued, a restart was not run).
> RATIS-1417 added a catch for this case on Ratis master in 2021, but it was
> removed again by RATIS-1481 before the next release, so no Ratis release used
> by Ozone has it.
> h3. Trigger
> The peer list is read from the configuration once, when the
> {{StorageContainerManager}} is constructed. A running SCM therefore does not
> know an SCM that was bootstrapped after it started:
> # Three SCMs with {{ozone.scm.ha.ratis.log.purge.enabled}} set to true.
> Bootstrap a fourth SCM and make it leader ({{ozone admin scm transfer}}, or
> an election).
> # One of the original followers, still running, falls behind the new leader's
> purged log (network partition, long pause) far enough that the purge passes
> its next index; the leader purges in steps of at least
> {{ozone.scm.ha.ratis.log.purge.gap}} (default 1000000 entries) (argued from
> code).
> # The new leader sends it an install snapshot notification. Its address is
> not in the follower's peer list, and the follower is stuck.
> With the default {{ozone.scm.ha.ratis.log.purge.enabled=false}} the leader
> purges only up to the lowest commit index of its peers, so a lagging follower
> keeps receiving entries and is not sent an install notification; a follower
> that is restarted instead loads the updated peer list (both argued from code,
> not run). An address mismatch between the configured peers and the Ratis peer
> address (HDDS-17205) leads to the same exception (code reading, not run).
> h3. Impact
> * The follower applies no further entries until it is restarted. Moving
> leadership back to an SCM it knows does not help.
> * The ring runs one replica short. In the four SCM ring of the trigger every
> remaining SCM is then needed for a quorum.
> No data is lost. A restart with the updated configuration should recover the
> SCM (argued, not run).
> h3. Reproduction
> [^TestBugMC3FollowerStuckAfterInstallNotification.java]
> ({{hadoop-ozone/integration-test}}, target path and Maven command in the
> class comment) follows the trigger on unmodified source, with purge enabled,
> a purge gap of 5 and small log segments: {{bootstrapSCM("scm4", true)}},
> leadership to scm4, one follower drops {{appendEntries}} and
> {{installSnapshot}} RPCs through Ratis' own {{CodeInjectionForTesting}} hook
> (a stand in for a partition, only timing) until scm4 has purged past the
> follower's next index, then the partition heals and leadership goes back to
> the original leader. It passes on unmodified source when the bug is present:
> {noformat}
> Leader commit 287, follower applied 53 -> 53 in 60s
> {noformat}
> The follower logged 12,483 "installation is in progress" refusals, and the
> stack of the one state machine call shows
> {{SCMStateMachine.notifyInstallSnapshotFromLeader}} (108s).
> h3. Patch
> [^MC-3-fail-install-snapshot-future-for-unknown-scm-leader.patch], against
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. It also applies to master at
> 257a71ab277.
> When no peer matches, {{notifyInstallSnapshotFromLeader}} returns a future
> that fails on the install snapshot executor with {{IOException("No SCM peer
> matches leader address ...")}} instead of throwing. Ratis then clears its
> in-progress index in {{whenComplete}}, the leader retries, and the follower
> installs the snapshot once the leader is an SCM it knows. The future fails on
> the executor rather than being returned already failed, because Ratis checks
> right after the call that the install is still in progress, and an already
> failed future makes that check throw on every retry. Nothing changes when the
> leader is found, and the existing "missing leader info" branch is unchanged.
> While an unknown SCM stays leader the follower still cannot install from it,
> and Ratis retries the notification without backoff. Each retry logs one ERROR
> line on the follower ("Failed to notify StateMachine to InstallSnapshot ...
> No SCM peer matches leader address ..."). In a local mini cluster that was
> 5269 lines in 30 seconds (about 175 per second); in 47 of those retries the
> future failed before Ratis' check, which then logged an
> {{IllegalStateException}} stack on the follower and a "Failed
> InstallSnapshot" warning on the leader. This lasts until leadership moves to
> an SCM the follower knows or the follower is restarted with the new
> configuration. Teaching a running SCM about new peers is a separate change.
> Covered by {{testInstallSnapshotNotificationFromUnknownLeaderFailsFuture}} in
> the existing {{TestSCMStateMachine}}, which also checks that a known leader
> is still used for the download. Without the change it fails with the
> {{IllegalStateException}}. With it {{TestSCMStateMachine}},
> {{TestSCMHAManagerImpl}} and {{TestSCMContext}} pass (23 tests), checkstyle
> is clean. With the patch the reproduction fails: the notifications from scm4
> fail, and after leadership moves back the follower installs the snapshot from
> the known leader and catches up to the leader's commit index ({{follower
> applied 285 -> 287}}, leader commit 287).
> HDDS-16932 and HDDS-16928 describe the same Ratis in-progress state on an SCM
> follower from other triggers (a leadership change during a successful
> download, a failed install after Ratis acknowledged it); they are related,
> not duplicates. The pending apache/ozone pull request for HDDS-17205 edits
> the same {{checkState}} line (adds a message). The two changes are
> independent; whichever lands second needs a one line rebase, and this patch
> uses the same message text.
> Found by TLA+ model checking and code review of the OM and SCM Ratis state
> machine integration (leader readiness, snapshot install and log failure
> handling) under HDDS-15926, on commit
> 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
