Indeed. That is bad. I cannot see a clean fix immediately, but we need to look at this.
Mind filing a ticket, Kireet? -- Lars ________________________________ From: Kireet <[email protected]> To: public-user-50Pas4EWwPEyzMRdD/[email protected] Sent: Friday, May 31, 2013 11:58 AM Subject: Re: HConnectionManager$HConnectionImplementation.locateRegionInMeta Even if I initiate the call via a pooled htable, the MetaScanner seems to use a concrete HTable instance. The constructor invoked seems to create a java ThreadPoolExecutor. I am not 100% sure but I think as long as nothing is submitted to the ThreadPoolExecutor it won't create any threads. I just wanted to confirm this was the case. I do see the connection is shared. --Kireet On 5/30/13 7:38 PM, Ted Yu wrote: > HTablePool$**PooledHTable is a wrapper around HTable. > > Here is how HTable obtains a connection: > > public HTable(Configuration conf, final byte[] tableName, final > ExecutorService pool) > throws IOException { > this.connection = HConnectionManager.getConnection(conf); > > Meaning the connection is a shared one based on certain key/value pairs > from conf. > > bq. So every call to batch will create a new thread? > > I don't think so. > > On Thu, May 30, 2013 at 11:28 AM, Kireet > <[email protected]> wrote: > >> >> >> Thanks, will give it a shot. So I should download 0.94.7 (latest stable) >> and run the patch tool on top with the backport? This is a little new to me. >> >> Also, I was looking at the stack below. From my reading of the code, the >> HTable.batch() call will always cause the prefetch call to occur, which >> will cause a new HTable object to get created. The constructor used in >> creating a new thread pool. So every call to batch will create a new >> thread? Or the HTable's thread pool never gets used as the pool is only >> used for writes? I think I am missing something but just want to confirm. >> >> Thanks >> Kireet >> >> On 5/30/13 12:48 PM, Himanshu Vashishtha wrote: >> >>> bq. Anoop attached backported patch in HBASE-8655. It should go into >>> >>> 0.94.9, the next release - current is 0.94.8 >>> >>> In case you want it sooner, you can apply 8655 patch and test/verify it. >>> >>> Thanks, >>> Himanshu >>> >>> >>> >>> On Thu, May 30, 2013 at 7:26 AM, Ted Yu <yuzhihong-** >>> Re5JQEeQqe8AvxtiuMwx3w@public.**gmane.org<yuzhihong-re5jqeeqqe8avxtiumwx3w-xmd5yjdbdmrexy1tmh2...@public.gmane.org>> >>> wrote: >>> >>> Anoop attached backported patch in HBASE-8655 >>>> >>>> It should go into 0.94.9, the next release - current is 0.94.8 >>>> >>>> Cheers >>>> >>>> On Thu, May 30, 2013 at 7:01 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+** >>>> [email protected] >>>> <kireet-teh5dpvpl8nqt0dzr%[email protected]>> >>>> wrote: >>>> >>>> >>>>> >>>>> How long do backports typically take? We have to go live in a month >>>>> ready >>>>> or not. Thanks for the quick replies Anoop and Ted. >>>>> >>>>> --Kireet >>>>> >>>>> >>>>> On 5/30/13 9:20 AM, Ted Yu wrote: >>>>> >>>>> 0.95 client is not compatible with 0.94 cluster. So you cannot use 0.95 >>>>>> client. >>>>>> >>>>>> Cheers >>>>>> >>>>>> On May 30, 2013, at 6:12 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+** >>>>>> AlfA-XMD5yJDbdMReXY1tMh2IBg@**public.gmane.org<alfa-xmd5yjdbdmrexy1tmh2ibg-xmd5yjdbdmrexy1tmh2...@public.gmane.org>< >>>>>> kireet-Teh5dPVPL8nQT0dZR%**2BAlfA-XMD5yJDbdMReXY1tMh2IBg@** >>>>>> public.gmane.org<kireet-teh5dpvpl8nqt0dzr%252balfa-xmd5yjdbdmrexy1tmh2ibg-xmd5yjdbdmrexy1tmh2...@public.gmane.org> >>>>>> >>>>>> >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> Would there be a problem if our cluster is 0.94 and we use a 0.95 >>>>>>> >>>>>> client? >>>> >>>>> >>>>>>> I am not familiar with the HBase code base, but I did a dump of the >>>>>>> thread that is actually running (below). It seems like it is related >>>>>>> >>>>>> to the >>>> >>>>> issue you mentioned as the running thread is doing the prefetch logic. >>>>>>> Would pre-splitting tables help here? We are doing some performance >>>>>>> >>>>>> tests >>>> >>>>> and essentially starting from an empty instance. >>>>>>> >>>>>>> java.lang.Thread.State: WAITING (on object monitor) >>>>>>> at java.lang.Object.wait(Native Method) >>>>>>> at java.lang.Object.wait(Object.****java:503) >>>>>>> at org.apache.zookeeper.****ClientCnxn.submitRequest(** >>>>>>> ClientCnxn.java:1309) >>>>>>> - locked <0x00000000e10cf830> (a org.apache.zookeeper.** >>>>>>> ClientCnxn$Packet) >>>>>>> at org.apache.zookeeper.****ZooKeeper.exists(ZooKeeper.****java:1036) >>>>>>> at org.apache.hadoop.hbase.****zookeeper.**** >>>>>>> RecoverableZooKeeper.exists(** >>>>>>> RecoverableZooKeeper.java:172) >>>>>>> at org.apache.hadoop.hbase.****zookeeper.ZKUtil.checkExists(**** >>>>>>> ZKUtil.java:450) >>>>>>> at org.apache.hadoop.hbase.****zookeeper.****ZooKeeperNodeTracker.** >>>>>>> checkIfBaseNodeAvailable(****ZooKeeperNodeTracker.java:208) >>>>>>> at org.apache.hadoop.hbase.****zookeeper.RootRegionTracker.** >>>>>>> waitRootRegionLocation(****RootRegionTracker.java:77) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****locateRegion(**** >>>>>>> HConnectionManager.java:874) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****locateRegionInMeta(** >>>>>>> HConnectionManager.java:987) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****locateRegion(**** >>>>>>> HConnectionManager.java:885) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****locateRegion(**** >>>>>>> HConnectionManager.java:846) >>>>>>> at org.apache.hadoop.hbase.****client.HTable.finishSetup(** >>>>>>> HTable.java:234) >>>>>>> at org.apache.hadoop.hbase.****client.HTable.<init>(HTable.**** >>>>>>> java:174) >>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(** >>>>>>> MetaScanner.java:160) >>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner.access$000(**** >>>>>>> MetaScanner.java:54) >>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner$1.connect(**** >>>>>>> MetaScanner.java:133) >>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner$1.connect(**** >>>>>>> MetaScanner.java:130) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager.** >>>>>>> execute(HConnectionManager.****java:383) >>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(** >>>>>>> MetaScanner.java:130) >>>>>>> at org.apache.hadoop.hbase.****client.MetaScanner.metaScan(** >>>>>>> MetaScanner.java:105) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****prefetchRegionCache(** >>>>>>> HConnectionManager.java:947) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****locateRegionInMeta(** >>>>>>> >>>>>>> HConnectionManager.java:1002) >>>>>>> - locked <0x00000000e882f778> (a java.lang.Object) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****locateRegion(**** >>>>>>> HConnectionManager.java:889) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****locateRegion(**** >>>>>>> HConnectionManager.java:846) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> HConnectionImplementation.****processBatchCallback(** >>>>>>> HConnectionManager.java:1533) >>>>>>> at org.apache.hadoop.hbase.****client.HConnectionManager$** >>>>>>> >>>>>>> HConnectionImplementation.****processBatch(**** >>>> HConnectionManager.java:1418) >>>> >>>>> at org.apache.hadoop.hbase.****client.HTable.batch(HTable.****java:702) >>>>>>> at org.apache.hadoop.hbase.****client.HTable.get(HTable.java:****671) >>>>>>> at org.apache.hadoop.hbase.****client.HTablePool$** >>>>>>> PooledHTable.get(HTablePool.****java:367) >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> Kireet >>>>>>> >>>>>>> On 5/30/13 12:16 AM, Anoop John wrote: >>>>>>> >>>>>>> Can you have a look at issue HBASE-8476? Seems related? A fix is >>>>>>>> available in HBASE-8346's patch.. >>>>>>>> >>>>>>>> -Anoop- >>>>>>>> >>>>>>>> On Thu, May 30, 2013 at 9:21 AM, Kireet <kireet-Teh5dPVPL8nQT0dZR+** >>>>>>>> AlfA-XMD5yJDbdMReXY1tMh2IBg@**public.gmane.org<alfa-xmd5yjdbdmrexy1tmh2ibg-xmd5yjdbdmrexy1tmh2...@public.gmane.org> >>>>>>>> < >>>>>>>> >>>>>>> kireet-Teh5dPVPL8nQT0dZR%**2BAlfA-XMD5yJDbdMReXY1tMh2IBg@** >>>> public.gmane.org<kireet-teh5dpvpl8nqt0dzr%252balfa-xmd5yjdbdmrexy1tmh2ibg-xmd5yjdbdmrexy1tmh2...@public.gmane.org> >>>>>> >>>> >>>> wrote: >>>>>>>> >>>>>>>> We are running hbase 0.94.6 in a concurrent environment and we are >>>>>>>> >>>>>>>>> seeing >>>>>>>>> the majority of our code stuck in this method at the synchronized >>>>>>>>> >>>>>>>> block >>>> >>>>> (line 997). Is there some configuration we can change to avoid it? It >>>>>>>>> seems >>>>>>>>> like we could disable prefetching, but that sounds like it could >>>>>>>>> >>>>>>>> cause >>>> >>>>> other problems? >>>>>>>>> >>>>>>>>> - org.apache.hadoop.hbase.******client.HConnectionManager$** >>>>>>>>> HConnectionImplementation.******locateRegionInMeta(byte[], byte[], >>>>>>>>> >>>>>>>>> byte[], >>>>>>>>> boolean, java.lang.Object, boolean) @bci=149, line=997 (Compiled >>>>>>>>> >>>>>>>> frame) >>>> >>>>> - org.apache.hadoop.hbase.******client.HConnectionManager$** >>>>>>>>> HConnectionImplementation.******locateRegion(byte[], byte[], >>>>>>>>> boolean, >>>>>>>>> >>>>>>>>> boolean) @bci=212, line=889 (Compiled frame) >>>>>>>>> - org.apache.hadoop.hbase.******client.HConnectionManager$** >>>>>>>>> HConnectionImplementation.******locateRegion(byte[], byte[]) >>>>>>>>> @bci=5, >>>>>>>>> line=846 >>>>>>>>> (Compiled frame) >>>>>>>>> - org.apache.hadoop.hbase.******client.HConnectionManager$** >>>>>>>>> >>>>>>>>> HConnectionImplementation.******processBatchCallback(java.****** >>>> util.List, >>>> >>>>> byte[], java.util.concurrent.******ExecutorService, java.lang.Object[], >>>>>>>>> org.apache.hadoop.hbase.******client.coprocessor.Batch$****** >>>>>>>>> Callback) >>>>>>>>> @bci=194, >>>>>>>>> line=1533 (Compiled frame) >>>>>>>>> - org.apache.hadoop.hbase.******client.HConnectionManager$** >>>>>>>>> HConnectionImplementation.******processBatch(java.util.List, >>>>>>>>> byte[], >>>>>>>>> java.util.concurrent.******ExecutorService, java.lang.Object[]) >>>>>>>>> >>>>>>>> @bci=30, >>>> >>>>> line=1418 (Compiled frame) >>>>>>>>> - org.apache.hadoop.hbase.******client.HTable.batch(java.util.** >>>>>>>>> ** >>>>>>>>> >>>>>>>>> **List) >>>>>>>>> @bci=24, line=702 (Compiled frame) >>>>>>>>> - org.apache.hadoop.hbase.******client.HTable.get(java.util.**** >>>>>>>>> **List) >>>>>>>>> @bci=2, >>>>>>>>> line=671 (Compiled frame) >>>>>>>>> - org.apache.hadoop.hbase.******client.HTablePool$** >>>>>>>>> PooledHTable.get(java.util.******List) @bci=5, line=367 (Compiled >>>>>>>>> >>>>>>>> frame) >>>> >>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
