> I have 6 node cassandra cluster all are setup with same > configuration I am getting fatal exceptions in one of the nodes > ERROR [Thread-604] 2011-03-29 20:19:13,218 AbstractCassandraDaemon.java > (line 114) Fatal exception in thread Thread[Thread-604,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [Thread-607] 2011-03-29 19:47:29,272 AbstractCassandraDaemon.java > (line 114) Fatal exception in thread Thread[Thread-607,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [Thread-605] 2011-03-29 19:38:09,081 AbstractCassandraDaemon.java > (line 114) Fatal exception in thread Thread[Thread-605,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR [MutationStage:2] 2011-03-29 19:37:16,659 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.lang.OutOfMemoryError: Java heap space > ERROR [GossipStage:1] 2011-03-29 20:27:29,898 AbstractCassandraDaemon.java > (line 114) Fatal exception in thread Thread[GossipStage:1,5,main] > java.lang.OutOfMemoryError: Java heap space > > All all the nodes have 32 G of ram. > > Everytime I try to restart the failed node I get the above errors.
Unless something is outright wrong, it sounds like you need to increase your JVM heap size in cassandra-env.sh. That you're getting it on start-up sounds consistent with commit log reply filling the heap in the form of memtables that are sized too big for your heap. There's a wiki page somewhere that describes the overall rule of thumb for heap sizing, but I can't find it right now. -- / Peter Schuller
