Siyao Meng created HDDS-17621:
---------------------------------
Summary: SCM brings a datanode back from DEAD without a full
container report, so it forgets the node's replicas for up to two hours
Key: HDDS-17621
URL: https://issues.apache.org/jira/browse/HDDS-17621
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-reregister-datanode-returning-from-dead.patch,
TestBugMC4ReturnedDeadNodeReplicasForgotten.java
h3. Mechanism
When a datanode goes DEAD, {{DeadNodeHandler}} removes its replicas from SCM
({{removeContainerReplicas}}), unless the node is in maintenance. If the
datanode was not restarted (network partition, long pause), it never registers
again: by default {{HeartbeatEndpointTask}} only counts the failed heartbeats,
and full container reports consumed by failed heartbeats are not put back. When
its heartbeats reach SCM again, {{SCMDatanodeHeartbeatDispatcher.dispatch}}
processes them normally (it sends {{ReregisterCommand}} only to unregistered
nodes and to STALE or DEAD nodes with an incompatible version), the health
check moves the node from DEAD to HEALTHY, and
{{UnhealthyToHealthyNodeHandler}} only notifies services. Nothing asks for a
full container report, so SCM learns the node's replicas again only from its
next periodic report, which comes every {{hdds.container.report.interval}} plus
a random delay of up to the same interval (60 to 120 minutes by default).
Incremental reports cover only containers whose state changes.
In that window the node is HEALTHY and in service, but SCM's replica set for
its containers does not include it. That set is used by:
* Target selection in ReplicationManager.
{{ReplicationManagerUtil.getExcludedAndUsedNodes}} only excludes nodes SCM
knows hold a replica, and it is shared by {{ECUnderReplicationHandler}},
{{RatisUnderReplicationHandler}}, {{MisReplicationHandler}} and
{{QuasiClosedStuckUnderReplicationHandler}}. The returned node can therefore be
chosen as a target for a container it already holds. On the datanode an EC
reconstruction then fails as a whole, because
{{ECReconstructionCoordinator.reconstructECContainerGroup}} first creates a
recovering container on every target and
{{KeyValueHandler.handleCreateContainer}} rejects it with
CONTAINER_ALREADY_EXISTS for any index. A Ratis replication is pushed by the
source and rejected by {{SendContainerRequestHandler}} with CONTAINER_EXISTS.
The existing replica is not touched. SCM is not told; the pending adds expire
after {{hdds.scm.replication.event.timeout}} (12 minutes) and the next round
picks targets again.
* Read pipelines for containers that are not open.
{{SCMClientProtocolServer.getContainerWithPipelineCommon}} builds them with
{{createPipelineForRead}} from the same replica set, so they leave the returned
node out.
h3. Trigger
# One or more datanodes lose contact with SCM, without a restart, for longer
than {{ozone.scm.dead.node.interval}} (10 minutes by default), and SCM removes
their replicas.
# Their heartbeats reach SCM again.
No client action is involved. If every holder of a container is away at the
same time (for example SCM cut off from the datanodes, or a network outage
covering all of a container's racks), SCM has no replica of that container at
all until the full reports arrive.
h3. Impact
* *Reads (correlated case):* when every holder of a closed container returned
from DEAD together, SCM returns a read pipeline with no nodes until the first
holder's next full report (run in the reproduction below), and
ReplicationManager reports the container MISSING meanwhile (argued from code).
What a client then sees was not run: OM fetches pipelines from SCM on a
location cache miss ({{ozone.om.container.location.cache.ttl}}, 360 minutes),
so a read that misses OM's cache in that window gets the empty pipeline and
cannot reach the data (argued from code).
* *Reads (single node):* the read pipeline lacks the returned node, so a Ratis
read has one replica fewer and an EC read may have to reconstruct the index
that node holds (argued from code).
* *Repair:* replication or reconstruction commands that cannot succeed are sent
to the returned node. With EC 3-2 on five nodes the returned node is the only
candidate, so every round fails until its full report (harmless there, the
index is present). If another index or replica is truly lost and the returned
node is among the few eligible targets, its repair is held up the same way, for
up to about two hours. With more spare nodes a round that draws k targets
includes the returned node with a probability of about k in the number of
eligible nodes, and each bad pick costs one event timeout. In the same window
SCM also rebuilds copies that are present on the returned node onto other
nodes, which later turns into over replication. Argued from code, not run here.
No data is deleted ({{EmptyContainerHandler}} does not act on a container with
no replicas, argued from code), and the state clears with the next full report.
h3. Reproduction
[^TestBugMC4ReturnedDeadNodeReplicasForgotten.java]
({{hadoop-hdds/server-scm}}, Maven command in the class comment) runs a real
in-process SCM ({{SCMNodeManager}} with health checks, {{DeadNodeHandler}} on
the SCM event queue, {{ContainerManager}}), the real
{{SCMDatanodeHeartbeatDispatcher}} and the client protocol server, with
heartbeats sent from the test and stale and dead intervals of 300 ms and 600
ms. A closed Ratis container has a replica on each of three datanodes. The
three stop heartbeating until they are DEAD and {{DeadNodeHandler}} has removed
the replicas, then heartbeat again: each reply carries no command, all three
become HEALTHY, SCM still lists no replica for the container, and
{{getContainerWithPipeline}} returns a pipeline with no nodes. A passing test
means the defect is present. It runs in about 2 seconds and does not start
datanodes.
h3. Patch
[^MC-4-reregister-datanode-returning-from-dead.patch], against
e4ae41fc4047c99af983020b4f33bd297c9d0ac5. It also applies to master at
3fc2387040f.
{{SCMDatanodeHeartbeatDispatcher.dispatch}} queues a {{ReregisterCommand}}
before processing a heartbeat from a node that SCM has as DEAD, so it is
returned in the reply to that heartbeat; the heartbeat itself is processed as
before. The datanode registers again at once, and registration carries a fresh
full container report, so SCM knows the node's replicas again within seconds
instead of after the next periodic report. This is the same command and
datanode path used after every SCM restart, and Recon already asks a returning
datanode to register again after a heartbeat gap
({{ReconNodeManager.processHeartbeat}}). There is no protocol, configuration or
datanode change. The existing parameterized test
{{TestSCMDatanodeHeartbeatDispatcher.testUnhealthyNodeWithCompatibleVersionProcessesHeartbeatNormally}}
now expects one re-register command for a DEAD node and none for a STALE one;
it fails without the change. The SCM node and server unit suites (256 tests)
pass and checkstyle is clean. With the patch the reproduction fails at its
first assertion because the reply contains the re-register command.
Each return of a DEAD datanode now causes one re-registration and one full
container report per SCM. Recon shares the dispatcher and already asks a
returning datanode to re-register; with this change it asks once more on the
following heartbeat. Narrow races remain and are no worse than today: a
registration report applied while {{DeadNodeHandler}} for the same node is
still pending can be undone by it (HDDS-17067), {{DeadNodeHandler}} clearing
the command queue between the two dispatcher steps would drop the command, and
a heartbeat that arrives while the health check is marking the node DEAD is not
asked.
Found by TLA+ model checking and code review of the SCM EC container
replication and placement repair paths under HDDS-15926, on commit
e4ae41fc4047c99af983020b4f33bd297c9d0ac5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
