i think i better make sure i understand how the row/key cache works. i
currently have both set to 10%. so if cassandra needs to read data from
an sstable that has 100 million rows, it will cache 10,000,000 rows of
data from that sstable? so if my row is ~4k, then we're looking at
~40gb used by cache?
Todd Burruss wrote:
the row/key cache is set to 10%, but memory usage stays good until an
anticompaction, hinted handoff, etc starts. (of course maybe i simply
don't pay attention to memory until something bad happens)
doesn't sound like anyone else is having trouble, so i'll keep review my
settings for cache, keep testing and try to get something more concrete.
Jonathan Ellis wrote:
it's almost certainly GC storming due to memory pressure. (matching
the thread dump / the threads using the CPU in top will confirm this.)
reducing your cache sizes might be the best option since you already
have a 44GB heap.
On Tue, Mar 16, 2010 at 12:17 PM, B. Todd Burruss <bburr...@real.com> wrote:
thx, i'll try that next time, already restarted node .. but i will say the
exact thing happened on another node as well.
Jonathan Ellis wrote:
You can still get a thread list w/ jstack, though.
On Tue, Mar 16, 2010 at 11:46 AM, Gary Dusbabek <gdusba...@gmail.com>
wrote:
On Tue, Mar 16, 2010 at 11:39, B. Todd Burruss <bburr...@real.com> wrote:
any other ideas on how to troubleshoot? i have tried kill -3 <java_pid>
in
the past but don't know where cassandra writes the console out. i'll
look
at scripts.
I have a sneaking suspicion that unless you're running with '-f' that
the thread dump goes into the ether. The only logical place to look
would be the log, which you've probably already done.
Gary.
