Hi Ted Sorry I forgot to mention, hbase-0.94.6 cdh 4.4.
Yeah, it was a pretty write intensive scenario that I think triggered it (importing a lot of datapoints into opentsdb). Do I flush the region manually using shell? Cheers, -Kristoffer On Sat, Mar 14, 2015 at 9:22 PM, Ted Yu <[email protected]> wrote: > Which release of HBase are you using ? > > I wonder if your cluster was hit with HBASE-10499. > > Cheers > > On Sat, Mar 14, 2015 at 1:13 PM, Kristoffer Sjögren <[email protected]> > wrote: > > > Hi > > > > It seems one of our region servers has been stuck closing a region for > > almost 22 hours. Puts or gets eventually fail with an exception [1]. > > > > Is there any safe way to release the region like restarting the region > > server? > > > > Cheers, > > -Kristoffer > > > > > > [1] > > > > 2015-03-14 21:02:24,316 INFO > org.apache.hadoop.hbase.regionserver.HRegion: > > Failed to unblock updates for region > > tsdb,\x00\x00\x9ETU\xAC@ > > > \x00\x00\x01\x00\x00\xAD\x00\x00\x05\x00\x00\xA7,1426282871862.4512f92b3d81e9142542d3b458223b63. > > 'IPC Server handler 9 on 60020' in 60000ms. The region is still busy. > > 2015-03-14 21:02:24,316 ERROR > > org.apache.hadoop.hbase.regionserver.HRegionServer: > > org.apache.hadoop.hbase.RegionTooBusyException: region is flushing > > at > > > > > org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:2731) > > at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:2002) > > at > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:2114) > > at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:606) > > at > > > > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) > > at > > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1428) > > >
