Dev Hingu created HBASE-29933:
---------------------------------
Summary: update_all_config hangs indefinitely when balancing event
is in progress
Key: HBASE-29933
URL: https://issues.apache.org/jira/browse/HBASE-29933
Project: HBase
Issue Type: Bug
Reporter: Dev Hingu
Assignee: Dev Hingu
update_all_config command hangs indefinitely if HMaster.balance() event is
going on.
When HMaster.balance() is running it acquires lock on CacheAwareLoadBalancer
and HMaster thread goes to sleep due to throttling in CacheAwareLoadBalancer.
Now, CacheAwareLoadBalancer.onConfigurationChange() waits to acquire the same
lock
Attaching stack traces for both thread
1. HMaster Thread :
{code:java}
#355 daemon prio=5 os_prio=0 cpu=2014.35ms elapsed=19554.89s
tid=0x00007f3fd018b310 nid=0x5fda waiting on condition [0x00007f3f6a3f1000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep([email protected]/Native Method)
at
org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer.throttle(CacheAwareLoadBalancer.java:197)
at
org.apache.hadoop.hbase.master.HMaster.executeRegionPlansWithThrottling(HMaster.java:2164)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:2122)
- locked <0x000000070b7d1438> (a
org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1998)
at
org.apache.hadoop.hbase.master.HMaster.balanceOrUpdateMetrics(HMaster.java:2010)
- locked <0x000000070b7d1438> (a
org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)
at
org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:47)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:161){code}
2. update configuration RPC thread :
{code:java}
#96 daemon prio=5 os_prio=0 cpu=523.03ms elapsed=19854.17s
tid=0x00007f3ffbb47ed0 nid=0x48a4 waiting for monitor entry
[0x00007f3f785fe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.master.balancer.BaseLoadBalancer.onConfigurationChange(BaseLoadBalancer.java:785)
- waiting to lock <0x000000070b7d1438> (a
org.apache.hadoop.hbase.master.balancer.CacheAwareLoadBalancer)
at
org.apache.hadoop.hbase.conf.ConfigurationManager.notifyAllObservers(ConfigurationManager.java:110)
- locked <0x000000070c9e2440> (a java.util.Collections$SetFromMap)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.updateConfiguration(HRegionServer.java:3927)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.updateConfiguration(RSRpcServices.java:3902)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
