The thrift deamon just stalls? You don't see this on 0.98.6 -- do you ever see retries exhausted exceptions on 0.98?
On Wed, Sep 2, 2015 at 5:17 PM, Abraham Tom <[email protected]> wrote: > Trying to see if anybody knows why this is happening and how to fix > we are on hbase 1.1.1 > > We use javascript to connect to hbase via thrift > every once in a while we get the following warning > > WARN [thrift-worker-3] thrift.ThriftServerRunner$HBaseHandler: Can't get > the location > This warning disconnects thrift from responding to any more requests > > full output from thrift log > thrift.ThriftServerRunner$HBaseHandler: Can't get the location > org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the > location > at > > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:309) > at > > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) > at > > org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61) > at > > org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) > at > > org.apache.hadoop.hbase.client.ReversedClientScanner.nextScanner(ReversedClientScanner.java:118) > at > > org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160) > at > org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155) > at > > org.apache.hadoop.hbase.client.ReversedClientScanner.<init>(ReversedClientScanner.java:56) > at > org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:800) > at > > org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.scannerOpenWithScan(ThriftServerRunner.java:1451) > at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > > org.apache.hadoop.hbase.thrift.HbaseHandlerMetricsProxy.invoke(HbaseHandlerMetricsProxy.java:67) > at com.sun.proxy.$Proxy11.scannerOpenWithScan(Unknown Source) > at > > org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4609) > at > > org.apache.hadoop.hbase.thrift.generated.Hbase$Processor$scannerOpenWithScan.getResult(Hbase.java:4593) > at > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > > org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478) > at org.apache.thrift.server.Invocation.run(Invocation.java:18) > at > org.apache.hadoop.hbase.thrift.CallQueue$Call.run(CallQueue.java:64) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: hconnection-0x538516ab closed > at > > org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1146) > at > > org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300) > ... 24 more > > jps shows that thrift is still operational, but the only way to clear this > up is to restart thrift. > Thrift is invoked as hbase-daemon.sh start thrift -hsha -f > hbase master and region look fine and hbase shell can still be invoked and > queried. > > > That invocation works on 0.98.6 instance and our javascript npm code base > has not changed > > > > > -- > Abraham Tom > Email: [email protected] > Phone: 415-515-3621 >
