The service keeps dieing at the same time every day and there is nothing in the app logs, it's going to be something external.
Sorry but I'm not sure what the problem with the memory usage is. Is the server running out of memory, or is it experiencing a lot of GC ? Cheers ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 11/10/2011, at 5:00 AM, Ashley Martens wrote: > I have check both the output file and the system log, neither have errors in > them. I don't believe anything external is killing the process, I could be > wrong but this node's setup is the same as all my other nodes (including > hardware) so it doesn't make much sense. > > > jsvc.exec -user cassandra -home /usr/lib/jvm/java-6-openjdk/jre/bin/../ > -pidfile /var/run/cassandra.pid -errfile &1 -outfile > /var/log/cassandra/output.log -cp > /usr/share/cassandra/antlr-3.1.3.jar:/usr/share/cassandra/apache-cassandra-0.7.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/avro-1.4.0-fixes.jar:/usr/share/cassandra/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/commons-cli-1.1.jar:/usr/share/cassandra/commons-codec-1.2.jar:/usr/share/cassandra/commons-collections-3.2.1.jar:/usr/share/cassandra/commons-lang-2.4.jar:/usr/share/cassandra/concurrentlinkedhashmap-lru-1.1.jar:/usr/share/casandra/guava-r05.jar:/usr/share/cassandra/high-scale-lib.jar:/usr/share/cassandra/jackson-core-asl-1.4.0.jar:/usr/share/cassandra/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra/jetty-6.1.21.jar:/usr/share/cassandra/jetty-util-6.1.21.jar:/usr/share/cassandra/jline-0.9.94.jar:/usr/share/cassandra/json-simple-1.1.jar:/usr/share/cassandra/jug-2.0.0.jar:/usr/share/cassandra/libthrift-0.5.jar:/usr/share/cassandra/log4j-1.2.16.jar:/usr/share/cassandra/servlet-api-2.5-20081211.jar:/usr/share/cassandra/slf4j-api-1.6.1.jar:/usr/share/cassandra/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/snakeyaml-1.6.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar > -Dlog4j.configuration=log4j-server.properties > -XX:HeapDumpPath=/var/lib/cassandra/java_1318260751.hprof > -XX:ErrorFile=/var/lib/casandra/hs_err_1318260751.log -ea > -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms24196M -Xmx24196M > -Xmn1600M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true > -Dcom.sun.management.jmxremote.port=8080 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > org.apache.cassandra.thrift.CassandraDaemon > > I have munin monitoring of JMX so when I talk about heap max then I'm > referring to: > > jmxObjectName java.lang:type=Memory > jmxAttributeName HeapMemoryUsage > jmxAttributeKey max > > The other crazy thing is the heap used is no where close to heap max. > > On Mon, Oct 10, 2011 at 12:40 AM, aaron morton <aa...@thelastpickle.com> > wrote: > Have you checked /var/log/cassandra/output.txt (the packaged install pipes > std out/err to there) or the system logs ? If there are no errors in the logs > it may well be something external killing it. > > With regard to memory usage, it's hard for people to help unless you provide > some numbers. What do you mean by MAX heap ? Is this the max used heap size > reported by JMX or the -Xmx setting passed to the server ? >
