Gargi-jais11 commented on code in PR #11322:
URL: https://github.com/apache/ozone/pull/11322#discussion_r4194944924
##########
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/diskbalancer/policy/DefaultContainerChoosingPolicy.java:
##########
@@ -91,62 +98,98 @@ public ContainerCandidate
chooseVolumesAndContainer(OzoneContainer ozoneContaine
return null;
}
- // Calculate ideal usage and threshold range (once)
- final double idealUsage = getIdealUsage(volumeUsages);
- final double actualThreshold = thresholdPercentage / 100.0;
- final double lowerThreshold = idealUsage - actualThreshold;
- final double upperThreshold = idealUsage + actualThreshold;
-
- if (LOG.isDebugEnabled()) {
- logVolumeBalancingState(volumeUsages, idealUsage, thresholdPercentage,
- lowerThreshold, upperThreshold, deltaMap);
+ // Balance each storage type independently so a container never crosses
types. Iterate
+ // StorageType.values() rather than the grouping keys to keep the order
stable across runs.
+ final Map<StorageType, List<VolumeFixedUsage>> usagesByStorageType =
volumeUsages.stream()
+ .collect(Collectors.groupingBy(v -> v.getVolume().getStorageType()));
+ for (StorageType storageType : StorageType.values()) {
Review Comment:
**DiskBalancerService#getTasks** calls the policy `availableTaskCount `times
per loop. Each call returns the first type that has a candidate. While SSD is
out of balance, every thread slot goes to SSD moves. DISK gets nothing until
SSD is within threshold, or until SSD has no movable container left.
For Example: a datanode has two small SSD volumes at 40% and 25%, and ten
large DISK volumes ranging from 95% down to 20%, with a 10% threshold. DISK is
the urgent group: its volumes are much larger and close to full. It still waits
for SSD to finish, and SSD moves can take hours when bandwidthInMB is
throttled. With parallelThread=5, all five slots go to SSD.
Might be we can discuss on this how we will handle. What I think right now
is:
- It should pick the group with the largest imbalance (highest.utilization -
idealUsage).
- Rotate the starting storage type on each call.
There might be some better options as well, I am exploring more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
