Siyao Meng created HDDS-17090:
---------------------------------
Summary: Follower SCM that has not applied finalization yet
rejects a datanode the leader already finalized, and the datanode shuts down
Key: HDDS-17090
URL: https://issues.apache.org/jira/browse/HDDS-17090
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-4-let-datanode-retry-when-non-leader-scm-rejects-registration.patch,
TestBugMC4LaggingFollowerFencesFinalizedDatanode.java
h3. Mechanism
Only the leader SCM sends {{FinalizeVersionCommand}} to datanodes, as soon as
it has applied FINALIZE itself
({{SCMNodeManager.sendFinalizeToDatanodeIfNeeded}}). It does not wait for the
other SCMs. A follower raises its apparent version only when it applies the
same entry, and its datanode protocol server keeps serving while it is behind:
{{SCMStateMachine.tryStartDNServerAndRefreshSafeMode}} starts it once the
follower reaches a fixed catch up index and never checks again.
When a finalized datanode registers with such a follower,
{{SCMNodeManager.shouldFenceDatanode}} finds the datanode's apparent version
above the follower's and {{register}} returns {{errorNodeNotPermitted}}.
{{RegisterEndpointTask.call}} moves that endpoint to SHUTDOWN, and
{{RunningDatanodeState.computeNextContainerState}} turns any SHUTDOWN endpoint
into a shutdown of the whole datanode. A datanode registers with every SCM, so
one lagging SCM is enough. Nothing in Ozone restarts the datanode after the
follower catches up.
HDDS-15482 intends this shutdown for a cluster that is genuinely not finalized.
Here the cluster is finalized and one SCM is only behind.
h3. Trigger
# SCM HA with all SCMs and datanodes pre-finalized.
# One follower stops receiving or applying new Raft log entries while datanodes
can still reach it (for example a partition between it and the leader, or a
stalled follower).
# Finalize. The leader and the other follower commit and apply FINALIZE, and
the leader finalizes every datanode on its next heartbeat.
# A finalized datanode restarts (ordinary operation, for example a rolling
restart) or re-registers, and registers with all SCMs.
Ordinary apply lag is not enough: without the cut off the follower applied
FINALIZE within milliseconds and nothing happened. The window is open for as
long as the follower stays behind.
By code reading only (not run): a follower installing a checkpoint keeps its
datanode server up and is pre-finalized until the install completes, so the
same rejection can happen during an install. At default config
({{ozone.scm.ha.ratis.log.purge.enabled=false}}) a running follower does not
need an install to catch up.
h3. Impact
Each finalized datanode that registers with the lagging follower stops, and
stays stopped after the follower has applied FINALIZE. Its replicas are
unavailable until it is restarted and are re-replicated once it is declared
dead. A rolling restart of datanodes during the window stops every restarted
datanode. No data loss was observed or is expected.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC4LaggingFollowerFencesFinalizedDatanode.java]
({{hadoop-ozone/integration-test}}) starts 3 SCMs and 3 datanodes, all
pre-finalized, and cuts one follower off from new log entries with the Ratis
{{CodeInjectionForTesting}} hook on {{RaftServerImpl.appendEntries}} (requests
with entries are rejected on that follower, heartbeats pass). It finalizes
through the public client, waits until the leader has finalized every datanode,
restarts one datanode, then lets the follower catch up. Output on the
unmodified code: the follower is at {{INITIAL_VERSION}} while all datanodes are
finalized, the restarted datanode logs "SCM rejected this datanode's
registration ... This datanode will shut down.", it is stopped, and it is still
stopped after the follower has applied FINALIZE. A passing test means the
defect is present.
h3. Suggested fix
A rejection from an SCM that is not the ready leader should not be final,
because that SCM may not have applied the leader's latest state. Alternatives
considered and not taken: letting the leader wait for all SCMs before sending
{{FinalizeVersionCommand}} (the leader does not know the followers' applied
index, and finalization would block while any SCM is down), letting followers
accept the registration (they would keep datanodes the leader rejects), and
deciding on the datanode (the register response does not say which SCM is the
leader, so this needs a new field).
h3. Patch
[^MC-4-let-datanode-retry-when-non-leader-scm-rejects-registration.patch],
against 7f9b6237deb2b06d73d1ac2b78cccaa60780c3ef. It also applies to master at
268654683b4.
{{SCMDatanodeProtocolServer.register}} throws {{ServerNotLeaderException}}
instead of returning {{errorNodeNotPermitted}} when this SCM is not the ready
leader ({{SCMContext.isLeaderReady}}, read before the registration decision is
made). The datanode handles an exception from register as a failed call and
retries on its next cycle, so it keeps running and registers with the follower
once that has caught up. The ready leader still returns
{{errorNodeNotPermitted}}, so a datanode that is incompatible with the cluster
is still shut down. The registration rules themselves are unchanged and still
applied on every SCM. There is no protocol change, the SCM RPC server already
suppresses logging of this exception, and Recon (no {{SCMContext}}) is not
affected.
This applies to every rejection reason on a non leader SCM, not only the
apparent version. Two outcomes change otherwise. A follower whose software is
older than a datanode that the leader accepts no longer shuts that datanode
down; it keeps running, registered with the leader. A datanode that would be
rejected but cannot get an answer from a ready leader (it reaches only
followers, the SCM quorum is lost, or a new leader is not ready yet) now keeps
running unregistered and retries until the ready leader rejects it, instead of
being shut down by a follower. A final shutdown then always comes from the SCM
whose decision is authoritative.
Covered by {{testOnlyReadyLeaderRejectsRegistrationAsFinal}} in the existing
{{TestSCMNodeManager}}: a pre-finalized SCM and a finalized datanode; a
follower and a leader that is not ready yet make the datanode retry, the ready
leader rejects it, and it is not registered. A pre-finalized datanode still
registers with the follower. Without the change it fails because the follower
returns {{errorNodeNotPermitted}}. With it {{TestSCMNodeManager}},
{{TestSCMStateMachine}}, {{TestSCMHAManagerImpl}} and {{TestSCMContext}} (90
tests) pass, and on a mini cluster {{TestScmHAFinalization}} (including
{{testFinalizedDatanodesShutDownWithPrefinalizedScm}}),
{{TestSCMDatanodeProtocolServer}} and
{{TestSCMFollowerCatchupWithContainerReport}} pass. Checkstyle is clean. With
the patch the reproduction fails at its first assertion: the restarted datanode
keeps running while the follower lags and registers with it after it has
applied FINALIZE.
Remaining window, by code reading only: a deposed leader that has not yet
stepped down still considers itself the ready leader and can still reject.
Found by TLA+ model checking and code review of the SCM upgrade finalization
paths under HDDS-15926, on commit 7f9b6237deb2b06d73d1ac2b78cccaa60780c3ef.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
