Wellington Chevreuil created HBASE-30300:
--------------------------------------------
Summary: CacheAwareLoadBalancer disproportionally favouring cache
ratio over skewness
Key: HBASE-30300
URL: https://issues.apache.org/jira/browse/HBASE-30300
Project: HBase
Issue Type: Bug
Reporter: Wellington Chevreuil
Assignee: Wellington Chevreuil
There’s a problem in the current CacheAwareLoadBalancer implementation where it
tends to disproportionally favour moves based on cache ratio, over skewness.
This happens even when cache usage is at capacity, and moving low cached
regions has close to no benefits (because the moved region wouldn't be cached
anyways, due to no cache space available).
Scanning the current CacheAwareLoadBalancer code with Claude Opus 4.6
identified the following issues behind this unexpected behaviour:
1 ) Generator weight mismatch in CacheAwareRegionSkewnessCostFunction and
CacheAwareCostFunction updateWeight(): Was writing to
LoadCandidateGenerator.class instead of the related implementations, causing
the generators to be picked randomly.
2) Skewness generator wasting iterations on old-server map:
CacheAwareSkewnessCandidateGenerator.pickRandomRegions() was draining
regionCacheRatioOnOldServerMap before doing skew-based moves. This could worsen
skew by moving regions to already-overloaded servers.
3) Throttle blocking skew moves for low-cache regions: throttle() is adding
sleep delays for regions that had near-zero cache ratio on their source server
(nothing to lose by moving quickly). Could be skipped when cacheRatioOnSource <
lowCacheRatioThreshold.
4) moveRegionToOldServer() disrupting well-cached regions: if a region is
already well-cached where it is, don't move it back to the "old" server based
on potentially stale historical data.
5) Prefetch overhead not subtracted from reported free space:
getServerBlockCacheFreeBytes() doesn't take into account acceptableFactor, when
calculating cache space available on potential destination servers. This can
cause moves that would then fail to prefetch in the target server.
6) Cumulative free space tracking across simulated moves: during plan
simulation, free space tracking is not updated correctly, and the the solver
could stack multiple regions onto one server without realizing it was filling
up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
