Since last night I am seeing CPU load spikes on our cassandra
boxes(Occasionally load goes up to 20, its a Amazon EC2 c1.xlarge with 300
iops EBS). After digging around a little I believe its related to heap
memory and flushing memtables.
>From logs:
WARN 03:22:03,414 Heap is 0.7786981388910019 full. You may need to reduce
memtable and/or cache sizes. Cassandra will now flush up to the two
largest memtables to free up memory. Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
WARN 03:22:03,415 Flushing CFS(Keyspace='XXX', ColumnFamily='XXXX') to
relieve memory pressure
I have three nodes and only 2 of them are hitting this high load, moreover
cluster is under extremely light load, no one is using it since yesterday
and I still see this load.
I also observed that `top -H` showed many threads in Sleep state and only a
handful in R state. `nodetool cfstat` showed the following for the
ColumnFamily in the above stated cassandra logs:
Column Family: XXXX
SSTable count: 8
Space used (live): 1479005837
Space used (total): 1479005837
Number of Keys (estimate): 2923008
Memtable Columns Count: 35375
Memtable Data Size: 7088479
Memtable Switch Count: 2393
Read Count: 2339668632
Read Latency: 3.042 ms.
Write Count: 360448535
Write Latency: 0.079 ms.
Pending Tasks: 0
Bloom Filter False Positives: 143197
Bloom Filter False Ratio: 0.73004
Bloom Filter Space Used: 7142048
Compacted row minimum size: 73
Compacted row maximum size: **785939**
Compacted row mean size: 1957
`Compacted Row maximum size` for other ColumFamily is significantly less
than this number.
When starting this cluster we set
> JVM_OPTS="$JVM_OPTS -Xss1000k"
We are using cassandra 1.1.0 and open-6-jdk
Can any one please help me understand whey with no load on the system I am
still seeing such high load on my machines.
Thanks
