sumitagrawl commented on code in PR #6050:
URL: https://github.com/apache/ozone/pull/6050#discussion_r1461545771
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerSelectionCriteria.java:
##########
@@ -88,19 +93,25 @@ private boolean
isContainerReplicatingOrDeleting(ContainerID containerID) {
* 5. Container should be closed.
* 6. If the {@link LegacyReplicationManager} is enabled, then the container
should not be an EC container.
* @param node DatanodeDetails for which to find candidate containers.
- * @return NavigableSet of candidate containers that satisfy the criteria.
+ * @return Set of candidate containers that satisfy the criteria.
*/
- public NavigableSet<ContainerID> getCandidateContainers(
+ public Set<ContainerID> getCandidateContainers(
DatanodeDetails node, long sizeMovedAlready) {
- NavigableSet<ContainerID> containerIDSet =
- new TreeSet<>(orderContainersByUsedBytes().reversed());
try {
- containerIDSet.addAll(nodeManager.getContainers(node));
+ // Initialize containerSet for node
+ if (!setMap.containsKey(node)) {
Review Comment:
I am trying to understand why do we need order based on containerUsed Size?
Do largest size container moving is efficient? or it can be random container
which can be moved?
we do not have statistics and always choose greatest container size only.
IMO, random can be better candidate in this for movement.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerSelectionCriteria.java:
##########
@@ -88,19 +93,25 @@ private boolean
isContainerReplicatingOrDeleting(ContainerID containerID) {
* 5. Container should be closed.
* 6. If the {@link LegacyReplicationManager} is enabled, then the container
should not be an EC container.
* @param node DatanodeDetails for which to find candidate containers.
- * @return NavigableSet of candidate containers that satisfy the criteria.
+ * @return Set of candidate containers that satisfy the criteria.
*/
- public NavigableSet<ContainerID> getCandidateContainers(
+ public Set<ContainerID> getCandidateContainers(
DatanodeDetails node, long sizeMovedAlready) {
- NavigableSet<ContainerID> containerIDSet =
- new TreeSet<>(orderContainersByUsedBytes().reversed());
try {
- containerIDSet.addAll(nodeManager.getContainers(node));
+ // Initialize containerSet for node
+ if (!setMap.containsKey(node)) {
+ addNodeToSetMap(node);
+ }
+ // In case the node is removed
+ nodeManager.getContainers(node);
} catch (NodeNotFoundException e) {
LOG.warn("Could not find Datanode {} while selecting candidate " +
"containers for Container Balancer.", node.toString(), e);
- return containerIDSet;
+ setMap.remove(node);
+ return Collections.emptySet();
}
+
+ NavigableSet<ContainerID> containerIDSet = setMap.get(node);
Review Comment:
A DN can have 1000s or more container sbased on size of disk at dn. So at
scale, this needs further optimized.
we do not need filter all containers for the criteria of size and others, it
can be filtered `dynamically on need basis`.
##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/container/balancer/ContainerBalancerSelectionCriteria.java:
##########
@@ -88,19 +93,25 @@ private boolean
isContainerReplicatingOrDeleting(ContainerID containerID) {
* 5. Container should be closed.
* 6. If the {@link LegacyReplicationManager} is enabled, then the container
should not be an EC container.
* @param node DatanodeDetails for which to find candidate containers.
- * @return NavigableSet of candidate containers that satisfy the criteria.
+ * @return Set of candidate containers that satisfy the criteria.
*/
- public NavigableSet<ContainerID> getCandidateContainers(
+ public Set<ContainerID> getCandidateContainers(
DatanodeDetails node, long sizeMovedAlready) {
- NavigableSet<ContainerID> containerIDSet =
- new TreeSet<>(orderContainersByUsedBytes().reversed());
try {
- containerIDSet.addAll(nodeManager.getContainers(node));
+ // Initialize containerSet for node
+ if (!setMap.containsKey(node)) {
+ addNodeToSetMap(node);
Review Comment:
can we add filtered container set to avoid re-filtering again when called ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
