i have a cassandra node that has stopped responding to nodetool tpstats
and not processing requests. memory is maxed on the JVM at 44gig. the
last i remember seeing from tpstats was over 150 hinted handoffs
pending. there hasn't been an OOM in the log yet, so not sure. JMX
isn't responding fast enough either to give stats.
since the JVM appears to be maxed on RAM i imagine GC is preventing
everything from working properly, but not sure why maxed. i have seen
memory max out from time to time during bootstrapping, moving tokens,
etc but can't get any stats when this happens because JVM isn't responding.
just want to throw this out to the list and see if anyone else has seen
issues with memory creeping up and eventually no more reponse from node.
any other ideas on how to troubleshoot? i have tried kill -3 <java_pid>
in the past but don't know where cassandra writes the console out. i'll
look at scripts.
thx
- cassandra not responding B. Todd Burruss
-