[
https://issues.apache.org/jira/browse/HDDS-16632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li reassigned HDDS-16632:
-----------------------------------
Assignee: Chu Cheng Li
> SCM loses a datanode's container list when the datanode re-registers with a
> new address, ports or version
> ---------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16632
> URL: https://issues.apache.org/jira/browse/HDDS-16632
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
>
> When a registered datanode registers again with a new IP address, hostname,
> ports or version, {{SCMNodeManager#register}} calls
> {{NodeStateManager#updateNode}}. {{NodeStateMap#updateNode}} then replaces
> the node's {{DatanodeEntry}} with a new one, which starts with an empty
> container list. A rolling upgrade changes the version of every datanode, so
> each restart during the upgrade takes this path, and so does any restart that
> changes a datanode's IP address.
> SCM uses this list to find replicas that disappeared from a datanode:
> * {{ContainerReportHandler}} compares each full container report with
> {{nodeManager.getContainers(dn)}}, and removes the replicas of containers
> that are in the list but not in the report.
> * {{DeadNodeHandler#removeContainerReplicas}} removes a dead datanode's
> replicas using the same list.
> So if a datanode loses a replica while it is down, for example on a failed
> disk, and then registers again with one of these changes, SCM never removes
> that replica. The report sent with the registration no longer includes it,
> and neither does the emptied list. SCM keeps counting the replica, so
> {{ReplicationManager}} does not treat the container as under-replicated. The
> replica is not removed when the datanode goes DEAD either.
> {{DatanodeAdminMonitorImpl#checkContainersReplicatedOnNode}} reads the same
> list. Until the container report sent with the registration has been
> processed, a datanode that is decommissioning or entering maintenance looks
> like it has no containers to wait for. This part was found by reading the
> code and has not been reproduced.
> A unit test with the real {{SCMNodeManager}} and {{ContainerReportHandler}}
> reproduces it:
> # Register a datanode and process a full container report from it with a
> CLOSED container that also has replicas on two other datanodes.
> # Register the datanode again with a different Ratis port.
> # Process full container reports from it that no longer include the container.
> SCM still lists 3 replicas, also after the datanode goes DEAD and
> {{DeadNodeHandler}} runs. Without the port change, the same reports bring it
> down to 2.
> Found while working on HDDS-16630.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
