[
https://issues.apache.org/jira/browse/HDDS-17090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17090:
------------------------------
Attachment: TestBugMC4LaggingFollowerFencesFinalizedDatanode.java
> Follower SCM that has not applied finalization yet rejects a datanode the
> leader already finalized, and the datanode shuts down
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17090
> URL: https://issues.apache.org/jira/browse/HDDS-17090
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-4-let-datanode-retry-when-non-leader-scm-rejects-registration.patch,
> TestBugMC4LaggingFollowerFencesFinalizedDatanode.java
>
>
> h3. Mechanism
> Only the leader SCM sends {{FinalizeVersionCommand}} to datanodes, as soon as
> it has applied FINALIZE itself
> ({{SCMNodeManager.sendFinalizeToDatanodeIfNeeded}}). It does not wait for the
> other SCMs. A follower raises its apparent version only when it applies the
> same entry, and its datanode protocol server keeps serving while it is
> behind: {{SCMStateMachine.tryStartDNServerAndRefreshSafeMode}} starts it once
> the follower reaches a fixed catch up index and never checks again.
> When a finalized datanode registers with such a follower,
> {{SCMNodeManager.shouldFenceDatanode}} finds the datanode's apparent version
> above the follower's and {{register}} returns {{errorNodeNotPermitted}}.
> {{RegisterEndpointTask.call}} moves that endpoint to SHUTDOWN, and
> {{RunningDatanodeState.computeNextContainerState}} turns any SHUTDOWN
> endpoint into a shutdown of the whole datanode. A datanode registers with
> every SCM, so one lagging SCM is enough. Nothing in Ozone restarts the
> datanode after the follower catches up.
> HDDS-15482 intends this shutdown for a cluster that is genuinely not
> finalized. Here the cluster is finalized and one SCM is only behind.
> h3. Trigger
> # SCM HA with all SCMs and datanodes pre-finalized.
> # One follower stops receiving or applying new Raft log entries while
> datanodes can still reach it (for example a partition between it and the
> leader, or a stalled follower).
> # Finalize. The leader and the other follower commit and apply FINALIZE, and
> the leader finalizes every datanode on its next heartbeat.
> # A finalized datanode restarts (ordinary operation, for example a rolling
> restart) or re-registers, and registers with all SCMs.
> Ordinary apply lag is not enough: without the cut off the follower applied
> FINALIZE within milliseconds and nothing happened. The window is open for as
> long as the follower stays behind.
> By code reading only (not run): a follower installing a checkpoint keeps its
> datanode server up and is pre-finalized until the install completes, so the
> same rejection can happen during an install. At default config
> ({{ozone.scm.ha.ratis.log.purge.enabled=false}}) a running follower does not
> need an install to catch up.
> h3. Impact
> Each finalized datanode that registers with the lagging follower stops, and
> stays stopped after the follower has applied FINALIZE. Its replicas are
> unavailable until it is restarted and are re-replicated once it is declared
> dead. A rolling restart of datanodes during the window stops every restarted
> datanode. No data loss was observed or is expected.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC4LaggingFollowerFencesFinalizedDatanode.java]
> ({{hadoop-ozone/integration-test}}) starts 3 SCMs and 3 datanodes, all
> pre-finalized, and cuts one follower off from new log entries with the Ratis
> {{CodeInjectionForTesting}} hook on {{RaftServerImpl.appendEntries}}
> (requests with entries are rejected on that follower, heartbeats pass). It
> finalizes through the public client, waits until the leader has finalized
> every datanode, restarts one datanode, then lets the follower catch up.
> Output on the unmodified code: the follower is at {{INITIAL_VERSION}} while
> all datanodes are finalized, the restarted datanode logs "SCM rejected this
> datanode's registration ... This datanode will shut down.", it is stopped,
> and it is still stopped after the follower has applied FINALIZE. A passing
> test means the defect is present.
> h3. Suggested fix
> A rejection from an SCM that is not the ready leader should not be final,
> because that SCM may not have applied the leader's latest state. Alternatives
> considered and not taken: letting the leader wait for all SCMs before sending
> {{FinalizeVersionCommand}} (the leader does not know the followers' applied
> index, and finalization would block while any SCM is down), letting followers
> accept the registration (they would keep datanodes the leader rejects), and
> deciding on the datanode (the register response does not say which SCM is the
> leader, so this needs a new field).
> h3. Patch
> [^MC-4-let-datanode-retry-when-non-leader-scm-rejects-registration.patch],
> against 7f9b6237deb2b06d73d1ac2b78cccaa60780c3ef. It also applies to master
> at 268654683b4.
> {{SCMDatanodeProtocolServer.register}} throws {{ServerNotLeaderException}}
> instead of returning {{errorNodeNotPermitted}} when this SCM is not the ready
> leader ({{SCMContext.isLeaderReady}}, read before the registration decision
> is made). The datanode handles an exception from register as a failed call
> and retries on its next cycle, so it keeps running and registers with the
> follower once that has caught up. The ready leader still returns
> {{errorNodeNotPermitted}}, so a datanode that is incompatible with the
> cluster is still shut down. The registration rules themselves are unchanged
> and still applied on every SCM. There is no protocol change, the SCM RPC
> server already suppresses logging of this exception, and Recon (no
> {{SCMContext}}) is not affected.
> This applies to every rejection reason on a non leader SCM, not only the
> apparent version. Two outcomes change otherwise. A follower whose software is
> older than a datanode that the leader accepts no longer shuts that datanode
> down; it keeps running, registered with the leader. A datanode that would be
> rejected but cannot get an answer from a ready leader (it reaches only
> followers, the SCM quorum is lost, or a new leader is not ready yet) now
> keeps running unregistered and retries until the ready leader rejects it,
> instead of being shut down by a follower. A final shutdown then always comes
> from the SCM whose decision is authoritative.
> Covered by {{testOnlyReadyLeaderRejectsRegistrationAsFinal}} in the existing
> {{TestSCMNodeManager}}: a pre-finalized SCM and a finalized datanode; a
> follower and a leader that is not ready yet make the datanode retry, the
> ready leader rejects it, and it is not registered. A pre-finalized datanode
> still registers with the follower. Without the change it fails because the
> follower returns {{errorNodeNotPermitted}}. With it {{TestSCMNodeManager}},
> {{TestSCMStateMachine}}, {{TestSCMHAManagerImpl}} and {{TestSCMContext}} (90
> tests) pass, and on a mini cluster {{TestScmHAFinalization}} (including
> {{testFinalizedDatanodesShutDownWithPrefinalizedScm}}),
> {{TestSCMDatanodeProtocolServer}} and
> {{TestSCMFollowerCatchupWithContainerReport}} pass. Checkstyle is clean. With
> the patch the reproduction fails at its first assertion: the restarted
> datanode keeps running while the follower lags and registers with it after it
> has applied FINALIZE.
> Remaining window, by code reading only: a deposed leader that has not yet
> stepped down still considers itself the ready leader and can still reject.
> Found by TLA+ model checking and code review of the SCM upgrade finalization
> paths under HDDS-15926, on commit 7f9b6237deb2b06d73d1ac2b78cccaa60780c3ef.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
