[
https://issues.apache.org/jira/browse/HDDS-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17621:
------------------------------
Attachment: MC-4-reregister-datanode-returning-from-dead.patch
> SCM brings a datanode back from DEAD without a full container report, so it
> forgets the node's replicas for up to two hours
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17621
> URL: https://issues.apache.org/jira/browse/HDDS-17621
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 0.5.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.2.1, 1.4.0, 2.0.0,
> 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1, 2.2.2
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-4-reregister-datanode-returning-from-dead.patch,
> TestBugMC4ReturnedDeadNodeReplicasForgotten.java
>
>
> h3. Mechanism
> When a datanode goes DEAD, {{DeadNodeHandler}} removes its replicas from SCM
> ({{removeContainerReplicas}}), unless the node is in maintenance. If the
> datanode was not restarted (network partition, long pause), it never
> registers again: by default {{HeartbeatEndpointTask}} only counts the failed
> heartbeats, and full container reports consumed by failed heartbeats are not
> put back. When its heartbeats reach SCM again,
> {{SCMDatanodeHeartbeatDispatcher.dispatch}} processes them normally (it sends
> {{ReregisterCommand}} only to unregistered nodes and to STALE or DEAD nodes
> with an incompatible version), the health check moves the node from DEAD to
> HEALTHY, and {{UnhealthyToHealthyNodeHandler}} only notifies services.
> Nothing asks for a full container report, so SCM learns the node's replicas
> again only from its next periodic report, which comes every
> {{hdds.container.report.interval}} plus a random delay of up to the same
> interval (60 to 120 minutes by default). Incremental reports cover only
> containers whose state changes.
> In that window the node is HEALTHY and in service, but SCM's replica set for
> its containers does not include it. That set is used by:
> * Target selection in ReplicationManager.
> {{ReplicationManagerUtil.getExcludedAndUsedNodes}} only excludes nodes SCM
> knows hold a replica, and it is shared by {{ECUnderReplicationHandler}},
> {{RatisUnderReplicationHandler}}, {{MisReplicationHandler}} and
> {{QuasiClosedStuckUnderReplicationHandler}}. The returned node can therefore
> be chosen as a target for a container it already holds. On the datanode an EC
> reconstruction then fails as a whole, because
> {{ECReconstructionCoordinator.reconstructECContainerGroup}} first creates a
> recovering container on every target and
> {{KeyValueHandler.handleCreateContainer}} rejects it with
> CONTAINER_ALREADY_EXISTS for any index. A Ratis replication is pushed by the
> source and rejected by {{SendContainerRequestHandler}} with CONTAINER_EXISTS.
> The existing replica is not touched. SCM is not told; the pending adds expire
> after {{hdds.scm.replication.event.timeout}} (12 minutes) and the next round
> picks targets again.
> * Read pipelines for containers that are not open.
> {{SCMClientProtocolServer.getContainerWithPipelineCommon}} builds them with
> {{createPipelineForRead}} from the same replica set, so they leave the
> returned node out.
> h3. Trigger
> # One or more datanodes lose contact with SCM, without a restart, for longer
> than {{ozone.scm.dead.node.interval}} (10 minutes by default), and SCM
> removes their replicas.
> # Their heartbeats reach SCM again.
> No client action is involved. If every holder of a container is away at the
> same time (for example SCM cut off from the datanodes, or a network outage
> covering all of a container's racks), SCM has no replica of that container at
> all until the full reports arrive.
> h3. Impact
> * *Reads (correlated case):* when every holder of a closed container returned
> from DEAD together, SCM returns a read pipeline with no nodes until the first
> holder's next full report (run in the reproduction below), and
> ReplicationManager reports the container MISSING meanwhile (argued from
> code). What a client then sees was not run: OM fetches pipelines from SCM on
> a location cache miss ({{ozone.om.container.location.cache.ttl}}, 360
> minutes), so a read that misses OM's cache in that window gets the empty
> pipeline and cannot reach the data (argued from code).
> * *Reads (single node):* the read pipeline lacks the returned node, so a
> Ratis read has one replica fewer and an EC read may have to reconstruct the
> index that node holds (argued from code).
> * *Repair:* replication or reconstruction commands that cannot succeed are
> sent to the returned node. With EC 3-2 on five nodes the returned node is the
> only candidate, so every round fails until its full report (harmless there,
> the index is present). If another index or replica is truly lost and the
> returned node is among the few eligible targets, its repair is held up the
> same way, for up to about two hours. With more spare nodes a round that draws
> k targets includes the returned node with a probability of about k in the
> number of eligible nodes, and each bad pick costs one event timeout. In the
> same window SCM also rebuilds copies that are present on the returned node
> onto other nodes, which later turns into over replication. Argued from code,
> not run here.
> No data is deleted ({{EmptyContainerHandler}} does not act on a container
> with no replicas, argued from code), and the state clears with the next full
> report.
> h3. Reproduction
> [^TestBugMC4ReturnedDeadNodeReplicasForgotten.java]
> ({{hadoop-hdds/server-scm}}, Maven command in the class comment) runs a real
> in-process SCM ({{SCMNodeManager}} with health checks, {{DeadNodeHandler}} on
> the SCM event queue, {{ContainerManager}}), the real
> {{SCMDatanodeHeartbeatDispatcher}} and the client protocol server, with
> heartbeats sent from the test and stale and dead intervals of 300 ms and 600
> ms. A closed Ratis container has a replica on each of three datanodes. The
> three stop heartbeating until they are DEAD and {{DeadNodeHandler}} has
> removed the replicas, then heartbeat again: each reply carries no command,
> all three become HEALTHY, SCM still lists no replica for the container, and
> {{getContainerWithPipeline}} returns a pipeline with no nodes. A passing test
> means the defect is present. It runs in about 2 seconds and does not start
> datanodes.
> h3. Patch
> [^MC-4-reregister-datanode-returning-from-dead.patch], against
> e4ae41fc4047c99af983020b4f33bd297c9d0ac5. It also applies to master at
> 3fc2387040f.
> {{SCMDatanodeHeartbeatDispatcher.dispatch}} queues a {{ReregisterCommand}}
> before processing a heartbeat from a node that SCM has as DEAD, so it is
> returned in the reply to that heartbeat; the heartbeat itself is processed as
> before. The datanode registers again at once, and registration carries a
> fresh full container report, so SCM knows the node's replicas again within
> seconds instead of after the next periodic report. This is the same command
> and datanode path used after every SCM restart, and Recon already asks a
> returning datanode to register again after a heartbeat gap
> ({{ReconNodeManager.processHeartbeat}}). There is no protocol, configuration
> or datanode change. The existing parameterized test
> {{TestSCMDatanodeHeartbeatDispatcher.testUnhealthyNodeWithCompatibleVersionProcessesHeartbeatNormally}}
> now expects one re-register command for a DEAD node and none for a STALE
> one; it fails without the change. The SCM node and server unit suites (256
> tests) pass and checkstyle is clean. With the patch the reproduction fails at
> its first assertion because the reply contains the re-register command.
> Each return of a DEAD datanode now causes one re-registration and one full
> container report per SCM. Recon shares the dispatcher and already asks a
> returning datanode to re-register; with this change it asks once more on the
> following heartbeat. Narrow races remain and are no worse than today: a
> registration report applied while {{DeadNodeHandler}} for the same node is
> still pending can be undone by it (HDDS-17067), {{DeadNodeHandler}} clearing
> the command queue between the two dispatcher steps would drop the command,
> and a heartbeat that arrives while the health check is marking the node DEAD
> is not asked.
> Found by TLA+ model checking and code review of the SCM EC container
> replication and placement repair paths under HDDS-15926, on commit
> e4ae41fc4047c99af983020b4f33bd297c9d0ac5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
