wenhao created HBASE-30405:
------------------------------
Summary: [Balancer] `HDFSBlocksDistribution.getTopHosts()` breaks
weight ties by hostname, causing the current RS to be misranked and occasional
zero-gain region moves
Key: HBASE-30405
URL: https://issues.apache.org/jira/browse/HBASE-30405
Project: HBase
Issue Type: Bug
Components: Balancer
Affects Versions: 2.5.11, 2.0.0
Reporter: wenhao
Assignee: wenhao
h2. Summary
When multiple hosts hold an equal total weight of a region's HFiles (i.e. equal
block locality), {{HDFSBlocksDistribution.getTopHosts()}} resolves the tie by
hostname. The resulting "most local host" ordering is therefore arbitrary with
respect to locality, and the host currently serving the region is frequently
not ranked first even though its locality is identical to the top host. This
contradicts the documented contract of {{BalancerClusterState.regionLocations}}
("regionIndex -> list of serverIndex sorted by locality") and, through the
most-local-entity selection used by {{{}LocalityBasedCandidateGenerator{}}},
can make the stochastic balancer emit zero-locality-gain move/swap proposals.
h2. Problem
{{HDFSBlocksDistribution.getTopHostsWithWeights()}} builds a {{TreeSet}}
ordered by {{HostAndWeight.WeightComparator}} and returns its
{{{}descendingSet(){}}}:
* {{{}WeightComparator.compare(l, r){}}}: primary key is weight (ascending);
*when weights are equal, the tie is broken by
{{l.getHost().compareTo(r.getHost())}}* (hostname, ascending).
* {{descendingSet()}} reverses that order, so the returned list is: weight
descending, and {*}among equal weights, hostname descending{*}.
Consequently the first element of {{getTopHosts()}} is the host with the
maximum block weight; if several hosts share the same maximum weight, the first
one is simply the lexicographically largest hostname — not the host with any
locality advantage.
h2. Root cause
# {{HDFSBlocksDistribution.java}}
** {{WeightComparator}} tie-break: {{return
l.getHost().compareTo(r.getHost());}}
** {{getTopHostsWithWeights()}} returns {{{}orderedHosts.descendingSet(){}}},
so equal-weight hosts are ordered by hostname {_}descending{_}.
# {{RegionLocationFinder.getTopBlockLocations(RegionInfo region)}} returns the
host list in exactly that order.
# {{BalancerClusterState.registerRegion()}} fills
{{regionLocations[regionIndex]}} from that list. The field contract says the
list is "sorted by locality", but under weight ties it is actually sorted by
hostname.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
