[
https://issues.apache.org/jira/browse/HBASE-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851743#comment-15851743
]
Ted Yu commented on HBASE-17565:
--------------------------------
Here was excerpt from test output for
TestStochasticLoadBalancer#testLargeCluster (with patch v2):
{code}
2017-02-03 16:13:16,986 INFO [main] balancer.StochasticLoadBalancer(362):
start StochasticLoadBalancer.balancer, initCost=107.0,
functionCost=RegionCountSkewCostFunction : (500.0, 0.2);
PrimaryRegionCountSkewCostFunction : (500.0, 0.0); MoveCostFunction : (7.0,
0.0); LocalityCostFunction : (0.0, 0.0); TableSkewCostFunction : (35.0, 0.2);
RegionReplicaHostCostFunction : (100000.0, 0.0); RegionReplicaRackCostFunction
: (10000.0, 0.0); ReadRequestCostFunction : (5.0, 0.0);
WriteRequestCostFunction : (5.0, 0.0); MemstoreSizeCostFunction : (5.0, 0.0);
StoreFileCostFunction : (5.0, 0.0);
2017-02-03 16:13:26,988 DEBUG [main] balancer.StochasticLoadBalancer(418):
Finished computing new load balance plan. Computation took 10006ms to try
1000000 different iterations. Found a solution that moves 19980 regions; Going
from a computed cost of 107.0 to a new cost of 8.8648
{code}
As you can see, though RegionReplicaHostCostFunction has high weight, its cost
was 0. The balancer was able to get to better overall cost by ignoring the
weight from RegionReplicaHostCostFunction.
> StochasticLoadBalancer may incorrectly skip balancing due to skewed
> multiplier sum
> ----------------------------------------------------------------------------------
>
> Key: HBASE-17565
> URL: https://issues.apache.org/jira/browse/HBASE-17565
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Priority: Critical
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17565.v1.txt, 17565.v2.txt
>
>
> I was investigating why a 6 node cluster kept skipping balancing requests.
> Here were the region counts on the servers:
> 449, 448, 447, 449, 453, 0
> {code}
> 2017-01-26 22:04:47,145 INFO
> [RpcServer.deafult.FPBQ.Fifo.handler=1,queue=0,port=16000]
> balancer.StochasticLoadBalancer: Skipping load balancing because balanced
> cluster; total cost is 127.0171157050385, sum multiplier is 111087.0 min cost
> which need balance is 0.05
> {code}
> The big multiplier sum caught my eyes. Here was what additional debug logging
> showed:
> {code}
> 2017-01-27 23:25:31,749 DEBUG
> [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
> balancer.StochasticLoadBalancer: class
> org.apache.hadoop.hbase.master.balancer.
> StochasticLoadBalancer$RegionReplicaHostCostFunction with multiplier 100000.0
> 2017-01-27 23:25:31,749 DEBUG
> [RpcServer.deafult.FPBQ.Fifo.handler=9,queue=0,port=16000]
> balancer.StochasticLoadBalancer: class
> org.apache.hadoop.hbase.master.balancer.
> StochasticLoadBalancer$RegionReplicaRackCostFunction with multiplier 10000.0
> {code}
> Note however, that no table in the cluster used read replica.
> I can think of two ways of fixing this situation:
> 1. If there is no read replica in the cluster, ignore the multipliers for the
> above two functions.
> 2. When cost() returned by the CostFunction is 0 (or very very close to 0.0),
> ignore the multiplier.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
