INFO [HINTED-HANDOFF-POOL:1] 2010-07-20 15:10:43,721 HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows to endpoint /10.129.28.23 ERROR [pool-1-thread-37895] 2010-07-20 15:10:51,622 CassandraDaemon.java (line 83) Uncaught exception in thread Thread[pool-1-thread-37895,5,main] java.lang.OutOfMemoryError: Java heap space at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:203) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1116) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)
The java version I am running java -version java version "1.6.0_18" Java(TM) SE Runtime Environment (build 1.6.0_18-b07) Java HotSpot(TM) 64-Bit Server VM (build 16.0-b13, mixed mode) The options I am running JVM_OPTS=" \ -ea \ -Xms1G \ -Xmx7G \ -XX:+UseParNewGC \ -XX:+UseConcMarkSweepGC \ -XX:+CMSParallelRemarkEnabled \ -XX:SurvivorRatio=8 \ -XX:MaxTenuringThreshold=1 \ -XX:+HeapDumpOnOutOfMemoryError \ -Dcom.sun.management.jmxremote.port=8181 \ -Dcom.sun.management.jmxremote.ssl=false \ -Dcom.sun.management.jmxremote.authenticate=false" ./nodetool --host localhost --port 8181 ring Address Status Load Range Ring 149416040858059761249215819391553456957 10.129.28.22 Down 8.28 GB 48330927683801271836312842096315901177 |<--| // server that crashed 10.129.28.20 Up 5.44 GB 115317719798097018520189339526900770000 | | 10.129.28.14 Up 1.53 GB 132300524084158455835439496737701548718 | | 10.129.28.23 Up 1.75 GB 149416040858059761249215819391553456957 |-->| Any idea, why setting the max heap space still forces a heap problem? Is there a problem with Garbage Collection? Should I restart my servers every few days?
