Hey JD, Thanks for the clarification. I also came across a previous thread which sort of talks about a similar problem. http://mail-archives.apache.org/mod_mbox/hbase-user/201204.mbox/%3ccagptdnfwnrsnqv7n3wgje-ichzpx-cxn1tbchgwrpohgcos...@mail.gmail.com%3E
I guess my problem is also similar to the fact that my writes are well distributed and at a given time I could be writing to a lot of regions. Some of the regions receive very little data but since the flush algorithm choose at random what to flush when "too many hlogs" is hit, it will flush a region with less than 10mb of data causing too many small files. This in-turn causes compaction storms where even though major compactions is disabled, some of the minor get upgraded to major and that's when things start getting worse. My compaction queues are still the same and so I doubt I will be coming out of this storm without bumping up max hlogs for now. Reducing regions per server is one option but then I will be wasting my resources since the servers at current load are at < 30% CPU and < 25% RAM. Maybe I can bump up heap space and give more memory to the the memstore. Sorry, I am just thinking out loud. Thanks, Viral On Thu, Jun 27, 2013 at 2:40 PM, Jean-Daniel Cryans <[email protected]>wrote: > No, all your data eventually makes it into the log, just potentially > not as quickly :) >
