sodonnel commented on code in PR #6558:
URL: https://github.com/apache/ozone/pull/6558#discussion_r1576126821
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/SCMCommonPlacementPolicy.java:
##########
@@ -206,11 +207,25 @@ public final List<DatanodeDetails> chooseDatanodes(
object of DatanodeDetails(with Topology Information) while trying to get the
random node from NetworkTopology should fix this. Check HDDS-7015
*/
+ List<DatanodeDetails> expandedExcludes =
expandExcludes(validateDatanodes(excludedNodes));
return chooseDatanodesInternal(validateDatanodes(usedNodes),
- validateDatanodes(excludedNodes), favoredNodes, nodesRequired,
+ expandedExcludes, favoredNodes, nodesRequired,
metadataSizeRequired, dataSizeRequired);
}
+ private List<DatanodeDetails> expandExcludes(List<DatanodeDetails> original)
{
+ Set<DatanodeDetails> expandedExcludes = new HashSet<>(original);
+ List<DatanodeDetails> list1 =
nodeManager.getNodes(NodeOperationalState.DECOMMISSIONING, null);
Review Comment:
We probably need to included the two maintenance states too, which means 4
iterations across all nodes in the cluster to find them. As this is a somewhat
hot path, especially for EC pipeline allocation I wonder if this will hurt
performance on larger clusters.
Partly this is due to the NodeManager interface - there is no method to find
all nodes not in_service in a single iteration, which is really what we want.
At the end of the selection, we have to check if the nodes are still good,
have enough space etc.
I wonder if we could do something smarter with the retry count to make it
try more times on a larger cluster?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
