Re: Help understanding aftermath of death by GC

2015-03-31 Thread Robert Coli
On Tue, Mar 31, 2015 at 9:12 AM, Jens Rantil wrote: > One issue when you are running a JVM and start running out of memory is > that the JVM can start throwing `OutOfMemoryError` in any thread - not > necessarily in the thread which is taking all the memory. I've seen this > happen multiple times

Re: Help understanding aftermath of death by GC

2015-03-31 Thread Jens Rantil
Hi Robert, On Tue, Mar 31, 2015 at 2:22 PM, Robert Wille wrote: > Can anybody help me understand why Cassandra wouldn’t recover? One issue when you are running a JVM and start running out of memory is that the JVM can start throwing `OutOfMemoryError` in any thread - not necessarily in the thr

Re: Help understanding aftermath of death by GC

2015-03-31 Thread Jason Wee
Hey Robert, you might want to start by looking into the statistics of cassandra, either exposed via nodetool or if you have monitoring system monitor the important metrics. I have read this article moment ago and I hope it help you http://aryanet.com/blog/cassandra-garbage-collector-tuning to begin

Help understanding aftermath of death by GC

2015-03-31 Thread Robert Wille
I moved my site over to Cassandra a few months ago, and everything has been just peachy until a few hours ago (yes, it would be in the middle of the night) when my entire cluster suffered death by GC. By death by GC, I mean this: [rwille@cas031 cassandra]$ grep GC system.log | head -5 INFO [Sch