Hi there, Recently I have been experiencing instability when scanning our HBASE cluster. The table we are trying to scan is 1.5B records 1TB, we have 12GB heap and 17 servers. Our GC options are as so: -XX:OnOutOfMemoryError=kill -9 %p -Xmx12000m -XX:+UseConcMarkSweepGC -Xmx12g -Xmx12g
The errors I am seeing in the logs: 2014-11-14 00:00:32,656 WARN org.apache.hadoop.hbase.util.Sleeper (regionserver60020): We slept 17355ms instead of 3000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired 2014-11-14 00:00:32,663 WARN org.apache.hadoop.hbase.util.Sleeper (regionserver60020.periodicFlusher): We slept 24734ms instead of 10000ms, this is likely due to a long garbage collecting pause and it's usually bad, see http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired We are using hbase.client.scanner.caching=1000. I suspect this may be a block cache issue. My question is if/how to disable the block cache for the scan queries? This is taking out writes and causing instability on the cluster. Thanks, Pere
