Thanks. I thought my problems may be related to the second leap and I ran sudo date -s "`date -u`" on all nodes. Things have improved much in the last 24 hours.
*Tamar Fraenkel * Senior Software Engineer, TOK Media [image: Inline image 1] ta...@tok-media.com Tel: +972 2 6409736 Mob: +972 54 8356490 Fax: +972 2 5612956 On Wed, Jul 4, 2012 at 1:39 PM, aaron morton <aa...@thelastpickle.com>wrote: > High CPU can be http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs > > memory usage looks ok http://wiki.apache.org/cassandra/FAQ#mmap > > Cheers > > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 3/07/2012, at 6:49 PM, Tamar Fraenkel wrote: > > Hi! > I have a Cassandra cluster on Amazon EC2 Datastax AMIs with 3 nodes and > replication factor of 2. > As of July 1st the cluster is very slow and seems to be loaded. > > Running top I get: > > top - 06:40:58 up 99 days, 21:30, 2 users, load average: 12.45, 13.37, > 14.01 > Tasks: 102 total, 1 running, 101 sleeping, 0 stopped, 0 zombie > Cpu(s): 21.0%us, 9.8%sy, 0.0%ni, 2.2%id, 0.0%wa, 0.4%hi, 0.4%si, > 66.3%st > Mem: 7647812k total, 7135752k used, 512060k free, 60668k buffers > Swap: 0k total, 0k used, 0k free, 4234008k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 18729 cassandr 20 0 8866m 2.5g 275m S 108 34.0 1415:37 jsvc > 21798 root 20 0 478m 217m 9760 S 44 2.9 1743:47 java > 3 root 20 0 0 0 0 S 20 0.0 564:59.09 ksoftirqd/0 > > > > These are the cassandra processes > *ps -ef | grep cassandra* > root 18727 1 0 Jul02 ? 00:00:00 jsvc.exec -user > cassandra-home /usr/lib/jvm/java-6-sun/jre/bin/../ -pidfile /var/run/ > cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra > /lib/avro-1.4.0-fixes.jar:/usr/share/cassandra > /lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra > /lib/commons-cli-1.1.jar:/usr/share/cassandra > /lib/commons-codec-1.2.jar:/usr/share/cassandra > /lib/commons-lang-2.4.jar:/usr/share/cassandra > /lib/compress-lzf-0.8.4.jar:/usr/share/cassandra > /lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra > /lib/guava-r08.jar:/usr/share/cassandra > /lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra > /lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra > /lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra > /lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/ > cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra > /lib/json-simple-1.1.jar:/usr/share/cassandra > /lib/libthrift-0.6.jar:/usr/share/cassandra > /lib/log4j-1.2.16.jar:/usr/share/cassandra > /lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra > /lib/slf4j-api-1.6.1.jar:/usr/share/cassandra > /lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra > /lib/snakeyaml-1.6.jar:/usr/share/cassandra > /lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra > /apache-cassandra-1.0.8.jar:/usr/share/cassandra > /apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra > /apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/ > java/commons-daemon.jar -Dlog4j.configuration=log4j-server.properties > -XX:HeapDumpPath=/var/lib/cassandra/java_1341216340.hprof > -XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log -ea > -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar > -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M > -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true > -Djava.rmi.server.hostname=10.34.158.33 > -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra > .thrift.CassandraDaemon > 108 18729 18727 99 Jul02 ? 23:26:48 jsvc.exec -user > cassandra-home /usr/lib/jvm/java-6-sun/jre/bin/../ -pidfile /var/run/ > cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp > /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra > /lib/avro-1.4.0-fixes.jar:/usr/share/cassandra > /lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra > /lib/commons-cli-1.1.jar:/usr/share/cassandra > /lib/commons-codec-1.2.jar:/usr/share/cassandra > /lib/commons-lang-2.4.jar:/usr/share/cassandra > /lib/compress-lzf-0.8.4.jar:/usr/share/cassandra > /lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/share/cassandra > /lib/guava-r08.jar:/usr/share/cassandra > /lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra > /lib/jackson-core-asl-1.4.0.jar:/usr/share/cassandra > /lib/jackson-mapper-asl-1.4.0.jar:/usr/share/cassandra > /lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/ > cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra > /lib/json-simple-1.1.jar:/usr/share/cassandra > /lib/libthrift-0.6.jar:/usr/share/cassandra > /lib/log4j-1.2.16.jar:/usr/share/cassandra > /lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra > /lib/slf4j-api-1.6.1.jar:/usr/share/cassandra > /lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra > /lib/snakeyaml-1.6.jar:/usr/share/cassandra > /lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra > /apache-cassandra-1.0.8.jar:/usr/share/cassandra > /apache-cassandra-thrift-1.0.8.jar:/usr/share/cassandra > /apache-cassandra.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/ > java/commons-daemon.jar -Dlog4j.configuration=log4j-server.properties > -XX:HeapDumpPath=/var/lib/cassandra/java_1341216340.hprof > -XX:ErrorFile=/var/lib/cassandra/hs_err_1341216341.log -ea > -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar > -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms1867M -Xmx1867M > -Xmn200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true > -Djava.rmi.server.hostname=10.34.158.33 > -Dcom.sun.management.jmxremote.port=7199 > -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra > .thrift.CassandraDaemon > > Attached is the cfstats from node #1. > Yesterday I restarted all the cluster, and it improved things a bit, but > still memory consumption is high and things are slower than they used to be > + many GC runs. > OpsCenter is running on node#1 and probes on other two nodes. > > Any help would be greatly appreciated! > Thanks, > > *Tamar Fraenkel * > Senior Software Engineer, TOK Media > > <tokLogo.png> > > > ta...@tok-media.com > Tel: +972 2 6409736 > Mob: +972 54 8356490 > Fax: +972 2 5612956 > > > > <cfstats4> > > >
<<tokLogo.png>>
