I am running a cassandra cluster of 6 nodes running RHEL6 virtualized by ESXi 5.0. Each VM is configured with 20GB of ram and 12 cores. Our test setup performs about 3000 inserts per second. The cassandra data partition is on a XFS filesystem mounted with options (noatime,nodiratime,nobarrier,logbufs=8). We have no swap enabled on the VMs and the vm.swappiness set to 0. To avoid any contention issues our cassandra VMs are not running any other application other than cassandra.
The test runs fine for about 12 hours or so. After that the performance starts to degrade to about 1500 inserts per sec. By 18-20 hours the inserts go down to 300 per sec. if i do a truncate, it starts clean, runs for a few hours (not as clean as rebooting). We find a direct correlation between kswapd kicking in after 12 hours or so and the performance degradation. If i look at the cached memory it is close to 10G. I am not getting a OOM error in cassandra. So looks like we are not running out of memory. Can some one explain if we can optimize this so that kswapd doesn't kick in. Our top output shows top - 16:23:54 up 2 days, 23:17, 4 users, load average: 2.21, 2.08, 2.02 Tasks: 213 total, 1 running, 212 sleeping, 0 stopped, 0 zombie Cpu(s): 1.6%us, 0.8%sy, 0.0%ni, 90.9%id, 6.3%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 20602812k total, 20320424k used, 282388k free, 1020k buffers Swap: 0k total, 0k used, 0k free, 10145516k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2586 root 20 0 36.3g 17g 8.4g S 32.1 88.9 8496:37 java java output root 2453 1 99 Sep30 pts/0 9-13:51:38 java -ea -javaagent:./apache-cassandra-0.8.6/bin/../lib/jamm-0.2.2.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms10059M -Xmx10059M -Xmn1200M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.19.104.14 -Djava.net.preferIPv4Stack=true -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -cp ./apache-cassandra-0.8.6/bin/../conf:./apache-cassandra-0.8.6/bin/../build/classes/main:./apache-cassandra-0.8.6/bin/../build/classes/thrift:./apache-cassandra-0.8.6/bin/../lib/antlr-3.2.jar:./apache-cassandra-0.8.6/bin/../lib/apache-cassandra-0.8.6.jar:./apache-cassandra-0.8.6/bin/../lib/apache-cassandra-thrift-0.8.6.jar:./apache-cassandra-0.8.6/bin/../lib/avro-1.4.0-fixes.jar:./apache-cassandra-0.8.6/bin/../lib/avro-1.4.0-sources-fixes.jar:./apache-cassandra-0.8.6/bin/../lib/commons-cli-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/commons-codec-1.2.jar:./apache-cassandra-0.8.6/bin/../lib/commons-collections-3.2.1.jar:./apache-cassandra-0.8.6/bin/../lib/commons-lang-2.4.jar:./apache-cassandra-0.8.6/bin/../lib/concurrentlinkedhashmap-lru-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/guava-r08.jar:./apache-cassandra-0.8.6/bin/../lib/high-scale-lib-1.1.2.jar:./apache-cassandra-0.8.6/bin/../lib/jackson-core-asl-1.4.0.jar:./apache-cassandra-0.8.6/bin/../lib/jackson-mapper-asl-1.4.0.jar:./apache-cassandra-0.8.6/bin/../lib/jamm-0.2.2.jar:./apache-cassandra-0.8.6/bin/../lib/jline-0.9.94.jar:./apache-cassandra-0.8.6/bin/../lib/json-simple-1.1.jar:./apache-cassandra-0.8.6/bin/../lib/libthrift-0.6.jar:./apache-cassandra-0.8.6/bin/../lib/log4j-1.2.16.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-examples.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-impl.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-jmx.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-remote.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-rimpl.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-rjmx.jar:./apache-cassandra-0.8.6/bin/../lib/mx4j-tools.jar:./apache-cassandra-0.8.6/bin/../lib/servlet-api-2.5-20081211.jar:./apache-cassandra-0.8.6/bin/../lib/slf4j-api-1.6.1.jar:./apache-cassandra-0.8.6/bin/../lib/slf4j-log4j12-1.6.1.jar:./apache-cassandra-0.8.6/bin/../lib/snakeyaml-1.6.jar org.apache.cassandra.thrift.CassandraDaemon Ring output [root@CAP4-CNode4 apache-cassandra-0.8.6]# ./bin/nodetool -h 127.0.0.1 ring Address DC Rack Status State Load Owns Token 141784319550391026443072753096570088105 10.19.104.11 datacenter1 rack1 Up Normal 19.92 GB 16.67% 0 10.19.104.12 datacenter1 rack1 Up Normal 19.3 GB 16.67% 28356863910078205288614550619314017621 10.19.104.13 datacenter1 rack1 Up Normal 18.57 GB 16.67% 56713727820156410577229101238628035242 10.19.104.14 datacenter1 rack1 Up Normal 19.34 GB 16.67% 85070591730234615865843651857942052863 10.19.105.11 datacenter1 rack1 Up Normal 19.88 GB 16.67% 113427455640312821154458202477256070484 10.19.105.12 datacenter1 rack1 Up Normal 20 GB 16.67% 141784319550391026443072753096570088105 [root@CAP4-CNode4 apache-cassandra-0.8.6]#
