The CMS compaction threshold is usually set to 75% as well, it might help to set it lower to 70% to see if that resolves these warnings as Cassandra will start CMS GC before it hits the 75% warning.
There is also a setting to lower the max amount of memory used for compacting each row. This may cause compactions to take longer though as each row that surpasses this limit will need to be compacted on disk instead of in memory. On Fri, Apr 19, 2013 at 9:00 AM, Michael Theroux <mthero...@yahoo.com>wrote: > Hello, > > We've recently upgraded from m1.large to m1.xlarge instances on AWS to > handle additional load, but to also relieve memory pressure. It appears to > have accomplished both, however, we are still getting a warning, 0-3 times > a day, on our database nodes: > > WARN [ScheduledTasks:1] 2013-04-19 14:17:46,532 GCInspector.java (line > 145) Heap is 0.7529240824406468 full. You may need to reduce memtable > and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold > in cassandra.yaml if you don't want Cassandra to do this automatically > > This is happening much less frequently than before the upgrade, but after > essentially doubling the amount of available memory, I'm curious on what I > can do to determine what is happening during this time. > > I am collecting all the JMX statistics. Memtable space is elevated but > not extraordinarily high. No GC messages are being output to the log. > > These warnings do seem to be occurring doing compactions of column > families using LCS with wide rows, but I'm not sure there is a direct > correlation. > > We are running Cassandra 1.1.9, with a maximum heap of 8G. > > Any advice? > Thanks, > -Mike -- Derek Williams
