> In our insert-tests the average heap usage is slowly growing up to the 3 GB > border (jconsole monitor over 50 min http://oi51.tinypic.com/k12gzd.jpg) and > the CompactionManger queue is also constantly growing up to about 50 jobs > pending.
Since you're obviously bottlenecking on compaction; are you building up lots of memtables flushes that don't complete? (I don't remember the name of the stage off hand, but it should be visible in cfstats). Also, if you simply stop writing suddenly and wait for the nodes to finish doing background activies, does memory usage go down again? (You may want to force a full GC before/after in order to do a proper test that is not affected by GC scheduling.) I don't remember the switches to JRockit, but you can definitely enable GC logging there which should tell you in more detail what's happening. IIRC, though possibly not for all GC modes, you should see periodic completions of concurrent GC:s that should collect all garbage that existed at the beginning of the GC cycle. Assuming you're not under so much load that this takes a very long time, that should give you a pretty good idea of the actual live set (which is probably going to be the low dips in your graphs, but it doesn't hurt confirming). -- / Peter Schuller
