[
https://issues.apache.org/jira/browse/HDDS-16633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li reassigned HDDS-16633:
-----------------------------------
Assignee: Chu Cheng Li
> Datanode can go missing from the network topology if it re-registers while
> SCM is removing it as DEAD
> -----------------------------------------------------------------------------------------------------
>
> Key: HDDS-16633
> URL: https://issues.apache.org/jira/browse/HDDS-16633
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
>
> When a datanode registers again with a new IP address, hostname, ports or
> version, {{SCMNodeManager#register}} replaces its object in the network
> topology with {{clusterMap.update}}, then replaces its {{DatanodeInfo}} with
> {{NodeStateManager#updateNode}}. The IP address and hostname case has always
> done this, and HDDS-16630 adds it for ports and version. This is not
> synchronized with the code that removes a DEAD datanode from the topology:
> * On master, {{DeadNodeHandler}} checks {{NetworkTopology#contains}}, then
> calls {{remove}}.
> * With HDDS-16623, {{NodeStateManager#checkNodesHealth}} marks the node DEAD,
> then removes it from the topology.
> If the re-registration runs between those two steps, the removal takes out
> the object that the re-registration just added, because
> {{InnerNodeImpl#remove}} removes whatever child has the node's name. The
> datanode ends up HEALTHY but missing from the topology, so placement never
> chooses it. It is only added back if it goes DEAD and recovers, or when SCM
> restarts. {{NetworkTopology#contains}} still returns true for its
> {{DatanodeInfo}}, because its parent pointer still points at the rack. On
> master, {{DeadNodeHandler}} also throws {{IllegalStateException}} from its
> {{Preconditions.checkState(node.getParent() == null)}} check.
> The window is small: the datanode has to register again at the moment SCM
> declares it DEAD. Both variants were reproduced in unit tests that run the
> re-registration from inside {{NetworkTopology#remove}}. Afterwards the node
> was HEALTHY, {{getNumOfLeafNode("")}} was 0, and {{contains()}} still
> returned true.
> Found while working on HDDS-16630.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
