> I posted mostly as a heads up for others using similar profiles (4GB
> heap on ~8GB boxes) to keep an eye out for. I expect a few people,
> particularly if they're on Amazon EC2, are running this type of setup.
>
> On the other hand, mum always said I was unique. ;)
So, now that I get that we have two different cases ;)
Re-reading your part of this thread, it seems you're not even having
OOM issues in Cassandra itself, right? You're just making (1) the
observation that memory graphs show little left, and (2) that nodetool
etc start failing on start with OOM?
So, to re-cap and/or add:
(1) Assuming you can live with the performance implications at least
temporarily, I suggest reconfiguring Cassandra for standard I/O mode
just to confirm that the JVM isn't significantly more 'resident' than
the maximum heap size you have configured.
(2) Enable GC logging with e.g. -Xloggc:/path/to/log and -XX:+PrintGC
-XX:+PrintGCDetails -XX:+PrintGCTimestamps.
(3) Observe top to confirm JVM memory use.
(4) Observe the +/- buffers in 'free' output. For example:
total used free shared buffers
cached
Mem: 3993448 3814416 179032 0 265380 2013404
-/+ buffers/cache: 1535632 2457816
Swap: 9928696 114248 9814448
The number I'm after above is 2457816 which is the amount of "free
memory if we pretend there is no page cache". Please make sure this is
the value you're getting, as some graphs might have weird definitions
of "free" memory or "cached". The 'free' command line tool is a known
entity.
One hypothesis: Is your data set slowly increasing, and the growth in
memory use over time is just a result of the data set being larger,
which would be reflected in the amount of data in page cache? If you
have no *other* significant activity that fills page cache (including
log files etc), page cache would be unused otherwise and then grow in
occupancy as your data set increases.
And to be clear: What is the actual negative observation that you have
made - do you see high latencies, saturation on disk I/O etc or is
this purely about seeming memory growth on graphs and observations of
memory usage by the JVM in top?
--
/ Peter Schuller
