Can it be https://issues.apache.org/jira/browse/HBASE-14196?
Do you idle your Thrift server for more than 10 min sometimes? -Vlad On Fri, Sep 18, 2015 at 4:23 PM, Nick Dimiduk <[email protected]> wrote: > The thrift deamon just stalls? You don't see this on 0.98.6 -- do you ever > see retries exhausted exceptions on 0.98? > > On Wed, Sep 2, 2015 at 5:17 PM, Abraham Tom <[email protected]> wrote: > > > Trying to see if anybody knows why this is happening and how to fix > > we are on hbase 1.1.1 > > > > We use javascript to connect to hbase via thrift > > every once in a while we get the following warning > > > > WARN [thrift-worker-3] thrift.ThriftServerRunner$HBaseHandler: Can't get > > the location > > This warning disconnects thrift from responding to any more requests > > > > full output from thrift log > > thrift.ThriftServerRunner$HBaseHandler: Can't get the location > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the > > location > > at > > > > > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:309) > > at > > > > > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) > > at > > > > > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61) > > at > > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) > > at > > > > > org.apache.hadoop.hbase.client.ReversedClientScanner.nextScanner(ReversedClientScanner.java:118) > > at > > > > > org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160) > > at > > > org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155) > > at > > > > > org.apache.hadoop.hbase.client.ReversedClientScanner.<init>(ReversedClientScanner.java:56) > > at > > org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:800) > > at > > > > > org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1451) > > at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at > > > > > org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67) > > at com.sun.proxy.$Proxy11.scannerOpenWithScan(Unknown Source) > > at > > > > > org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4609) > > at > > > > > org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4593) > > at > > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > > at > org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > > at > > > > > org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478) > > at org.apache.thrift.server.Invocation.run(Invocation.java:18) > > at > > org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64) > > at > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.io.IOException: hconnection-0x538516ab closed > > at > > > > > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1146) > > at > > > > > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) > > ... 24 more > > > > jps shows that thrift is still operational, but the only way to clear > this > > up is to restart thrift. > > Thrift is invoked as hbase-daemon.sh start thrift -hsha -f > > hbase master and region look fine and hbase shell can still be invoked > and > > queried. > > > > > > That invocation works on 0.98.6 instance and our javascript npm code base > > has not changed > > > > > > > > > > -- > > Abraham Tom > > Email: [email protected] > > Phone: 415-515-3621 > > >
