BTW, thanks Michael. I am surprised why I didn't search for Cassandra oom before. I got some good links that discuss that. Will try to optimize and see how it goes.
On Wed, Sep 24, 2014 at 10:27 PM, Prem Yadav <ipremya...@gmail.com> wrote: > Well its not the Linux OOM killer. The system is running with all default > settings. > > Total memory 7GB- Cassandra gets assigned 2GB > 2 core processors. > Two rings with 3 nodes in each ring. > > On Wed, Sep 24, 2014 at 9:53 PM, Michael Shuler <mich...@pbandjelly.org> > wrote: > >> On 09/24/2014 11:32 AM, Prem Yadav wrote: >> >>> this is an issue that has happened a few times. We are using DSE 4.0 >>> >> >> I believe this is Apache Cassandra 2.0.5, which is better info for this >> list. >> >> One of the Cassandra nodes is detected as dead by the opscenter even >>> though I can see the process is up. >>> >>> the logs show heap space error: >>> >>> INFO [RMI TCP Connection(18270)-172.31.49.189] 2014-09-24 08:31:05,340 >>> StorageService.java (line 2538) Starting repair command #30766, >>> repairing 1 ranges for keyspace <keyspace> >>> ERROR [BatchlogTasks:1] 2014-09-24 08:48:54,780 CassandraDaemon.java >>> (line 196) Exception in thread Thread[BatchlogTasks:1,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> at java.util.ArrayList.<init>(Unknown Source) >>> >> >> OOM. >> >> System environment and configuration modification details might be >> helpful for others to give you advice. Searching for "cassandra oom" gave >> me a few good links to read, and knowing some details about your nodes >> might be really helpful. Additionally, CASSANDRA-7507 [0] suggests that an >> OOM leaving the process running in an unclean state is not desired, and the >> process should be killed. >> >> Several of the search links provide details on how to capture and dig >> around a heap dump to aid in troubleshooting. >> >> [0] https://issues.apache.org/jira/browse/CASSANDRA-7507 >> -- >> Kind regards, >> Michael >> > >
