Didn't have time yet to properly respond to this one, but check out the "Very high memory utilization (not caused by mmap on sstables)" thread from today. You're using the same Ubuntu/JVM, right? Seems to match your observations.
(Sorry no ML archive link; the ML archives for Apache projects has never worked properly for me in my browsers... too broken-ajaxy.) -- / Peter Schuller