We faced same issue sometimes too. 1) Linux OOM killer kill your Cassandra process. You should find this event log in /var/log/message.2) The JVM crashed. You should be able to find the hs_err_pid file under /tmp folder, if you didn't specify the location when you started your JVM. We still cannot find out the root reason why the JVM crashed. The core dump file indicates the JVM crashed in compile Java code, several times though. It didn't help even we upgraded from latest JDK 1.6 to latest JDK 1.7. Yong
> From: dean.hil...@nrel.gov > To: user@cassandra.apache.org > Date: Wed, 18 Sep 2013 06:59:07 -0600 > Subject: Re: cassandra just gone..no heap dump, no log info > > Swappiness is set to 60 though cassandra recommendations are to turn swap > completely off(we still have not done that as far as I know though) and sure > enough linux killed it. > > Dean > > From: Juan Manuel Formoso <jform...@gmail.com<mailto:jform...@gmail.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Date: Wednesday, September 18, 2013 6:57 AM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> > Subject: Re: cassandra just gone..no heap dump, no log info > > This shouldn't happen if you have swap active in the server > > On Wednesday, September 18, 2013, Franc Carter wrote: > > A random guess - possibly an OOM (Out of Memory) where Linux will kill a > process to recover memory when it is desperately low on memory. Have a look > in either your syslog output of the output of dmesg > > cheers > > > On Wed, Sep 18, 2013 at 10:21 PM, Hiller, Dean > <dean.hil...@nrel.gov<javascript:_e({},%20'cvml',%20'dean.hil...@nrel.gov');>> > wrote: > Anyone know how to debug cassandra processes just exiting? There is no info > in the cassandra logs and there is no heap dump file(which in the past has > shown up in /opt/cassandra/bin directory for me). > > This occurs when running a map/reduce job that put severe load on the system. > The logs look completely fine. I find it odd > > 1. No logs of why it exited at all > 2. No heap dump which would imply there would be no logs as it crashed > > Is there any other way a process can die and linux would log it somehow? > (like running out of memory) > > Thanks, > Dean > > > > -- > > Franc Carter| Systems architect|Sirca Ltd > <javascript:_e({},%20'cvml',%20'marc.zianideferra...@sirca.org.au');> > > franc.car...@sirca.org.au<javascript:_e({},%20'cvml',%20'franc.car...@sirca.org.au');> > | www.sirca.org.au<http://www.sirca.org.au/> > > Tel: +61 2 8355 2514 > > Level 4, 55 Harrington St, The Rocks NSW 2000 > > PO Box H58, Australia Square, Sydney NSW 1215 > > > > -- > Juan Manuel Formoso > Senior Geek > http://twitter.com/juanformoso > http://seniorgeek.com.ar<http://seniorgeek.com.ar/> > LLAP
