Hi,
We are using Cassandra 0.6.12 in a cluster of 9 nodes. Each node is
64-bit, has 4 cores and 4G of RAM and runs on Ubuntu Lucid with the
stock 2.6.32-31-generic kernel. We use the Sun/Oracle JDK.
Here's the problem: The Cassandra process starts up with 1.1G resident
memory (according to top) but slowly grows to 2.1G at a rate that
seems proportional to the write load. No writes, no growth. The node
is running other memory-sensitive applications (a second JVM for our
in-house webapp and a short-lived C++ program) so we need to ensure
that each process stays within certain bounds as far as memory
requirements go. The nodes OOM and crash when the Cassandra process is
at 2.1G so I can't say if the growth is bounded or not.
Looking at the /proc/$pid/smaps for the Cassandra process it seems to
me that it is the native heap of the Cassandra JVM that is leaking. I
attached a readable version of the smaps file generated by [1].
Some more data: Cassandra runs with default command line arguments,
which means it gets 1G heap. The JNA jar is present and Cassandra logs
that the memory locking was successful. In storage-conf.xml,
DiskAccessMode is mmap_index_only. Other than that and some increased
timeouts we left the defaults. Swap is completely disabled. I don't
think this is related but I am mentioning it anyways: overcommit [2]
is always-on (vm.overcommit_memory=1). Without that we get OOMs when
our application JVM is fork()'ing and exec()'ing our C++program even
though there is enough free RAM to satisfy the demands of the C++
program. We think this is caused by a flawed kernel heuristic that
assumes that the forked process (our C++ app) is as big as the forking
one (the 2nd JVM). Anyways, the Cassandra process leaks with both,
vm.overcommit_memory=0 (the default) and 1.
Whether it is the native heap that leaks or something else, I think
that 1.1G of additional RAM for 1G of Java heap can't be normal. I'd
be grateful for any insights or pointers at what to try next.
[1] http://bmaurer.blogspot.com/2006/03/memory-usage-with-smaps.html
[2] http://www.win.tue.nl/~aeb/linux/lk/lk-9.html#ss9.6
VMSIZE: 2205292 kb
RSS: 1756732 kb total
4608 kb shared
1116 kb private clean
1751004 kb private dirty
PRIVATE MAPPINGS
vmsize rss clean rss dirty file
1079104 kb 0 kb 1066904 kb
438656 kb 0 kb 438288 kb [heap]
66280 kb 0 kb 66280 kb
65704 kb 0 kb 65704 kb
56516 kb 0 kb 42736 kb
60848 kb 0 kb 30496 kb
61756 kb 0 kb 16928 kb
8104 kb 0 kb 8104 kb
6592 kb 0 kb 5400 kb
1616 kb 0 kb 564 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/server/libjvm.
208 kb 0 kb 208 kb
160 kb 0 kb 160 kb
228 kb 0 kb 156 kb
1016 kb 0 kb 100 kb
88 kb 0 kb 88 kb
1016 kb 0 kb 76 kb
1016 kb 0 kb 76 kb
1016 kb 0 kb 76 kb
1016 kb 0 kb 76 kb
1016 kb 0 kb 76 kb
1016 kb 0 kb 76 kb
1016 kb 0 kb 76 kb
1016 kb 0 kb 76 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1016 kb 0 kb 72 kb
1012 kb 0 kb 72 kb [stack]
1016 kb 0 kb 64 kb
1016 kb 0 kb 64 kb
1016 kb 0 kb 64 kb
1016 kb 0 kb 56 kb
1016 kb 0 kb 52 kb
1016 kb 0 kb 52 kb
1016 kb 0 kb 48 kb
1016 kb 0 kb 48 kb
1016 kb 0 kb 48 kb
1016 kb 0 kb 48 kb
1016 kb 0 kb 48 kb
1016 kb 0 kb 48 kb
1016 kb 0 kb 48 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 44 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 40 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 36 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 32 kb
1016 kb 0 kb 24 kb
1016 kb 0 kb 24 kb
1016 kb 0 kb 24 kb
1016 kb 0 kb 24 kb
1016 kb 0 kb 24 kb
1016 kb 0 kb 24 kb
1016 kb 0 kb 24 kb
28 kb 0 kb 20 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libjava.so
20 kb 0 kb 20 kb
32 kb 4 kb 16 kb /tmp/hsperfdata_cassandra/1718
12 kb 0 kb 12 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libnet.so
1024 kb 0 kb 12 kb
1024 kb 0 kb 12 kb
1024 kb 0 kb 12 kb
1024 kb 0 kb 12 kb
12 kb 0 kb 12 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libzip.so
12 kb 0 kb 12 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libverify.so
16 kb 0 kb 12 kb
8 kb 0 kb 8 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libnio.so
1024 kb 0 kb 8 kb
8 kb 0 kb 8 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libmanagement.
1016 kb 0 kb 8 kb
1016 kb 0 kb 8 kb
1016 kb 0 kb 8 kb
1024 kb 0 kb 8 kb
8 kb 0 kb 8 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/native_threads
4 kb 0 kb 4 kb /usr/bin/jsvc
4 kb 0 kb 4 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/librmi.so
4 kb 0 kb 4 kb /lib/libresolv-2.11.1.so
4 kb 0 kb 4 kb /lib/libresolv-2.11.1.so
8 kb 0 kb 4 kb
4 kb 0 kb 4 kb /lib/libnss_dns-2.11.1.so
4 kb 0 kb 4 kb /lib/libnss_dns-2.11.1.so
4 kb 0 kb 4 kb
4 kb 0 kb 4 kb
4 kb 0 kb 4 kb /lib/librt-2.11.1.so
4 kb 0 kb 4 kb /lib/librt-2.11.1.so
4 kb 0 kb 4 kb /lib/libm-2.11.1.so
4 kb 0 kb 4 kb /lib/libm-2.11.1.so
4 kb 0 kb 4 kb /lib/libnss_files-2.11.1.so
4 kb 0 kb 4 kb /lib/libnss_nis-2.11.1.so
4 kb 0 kb 4 kb /lib/libnss_compat-2.11.1.so
4 kb 0 kb 4 kb /lib/libc-2.11.1.so
4 kb 0 kb 4 kb /lib/libpthread-2.11.1.so
16 kb 0 kb 4 kb
4 kb 0 kb 4 kb /lib/libdl-2.11.1.so
16 kb 0 kb 4 kb
4 kb 0 kb 4 kb /lib/ld-2.11.1.so
4 kb 0 kb 4 kb /lib/ld-2.11.1.so
4 kb 0 kb 4 kb
36 kb 4 kb 0 kb /usr/bin/jsvc
4 kb 4 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/librmi.so
624 kb 72 kb 0 kb
/var/lib/cassandra/data/ps-main-0/Scores-341-Index.db
20 kb 12 kb 0 kb
/var/lib/cassandra/data/ps-main-0/Scores-350-Index.db
12 kb 12 kb 0 kb
/var/lib/cassandra/data/ps-main-0/Scores-355-Index.db
4 kb 4 kb 0 kb
/var/lib/cassandra/data/ps-main-0/Scores-356-Index.db
28 kb 12 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libnio.so
88 kb 4 kb 0 kb /lib/libresolv-2.11.1.so
76 kb 12 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libnet.so
24 kb 12 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libmanagement.
1632 kb 80 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/rt.jar
56 kb 24 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libzip.so
28 kb 4 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/native_threads
164 kb 40 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libjava.so
28 kb 4 kb 0 kb /lib/librt-2.11.1.so
8440 kb 756 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/server/libjvm.
40 kb 4 kb 0 kb /lib/libnss_nis-2.11.1.so
92 kb 8 kb 0 kb /lib/libnsl-2.11.1.so
32 kb 4 kb 0 kb /lib/libnss_compat-2.11.1.so
1512 kb 28 kb 0 kb /lib/libc-2.11.1.so
128 kb 8 kb 0 kb /lib/ld-2.11.1.so
52 kb 4 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libverify.so
SHARED MAPPINGS
vmsize rss clean rss dirty file
8440 kb 3668 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/server/libjvm.
1512 kb 468 kb 0 kb /lib/libc-2.11.1.so
128 kb 84 kb 0 kb /lib/ld-2.11.1.so
96 kb 68 kb 0 kb /lib/libpthread-2.11.1.so
164 kb 56 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libjava.so
88 kb 48 kb 0 kb /lib/libresolv-2.11.1.so
76 kb 36 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libnet.so
28 kb 16 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libnio.so
20 kb 16 kb 0 kb /lib/libnss_dns-2.11.1.so
28 kb 16 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/native_threads
48 kb 16 kb 0 kb /lib/libnss_files-2.11.1.so
16 kb 0 kb 16 kb /lib/libc-2.11.1.so
24 kb 12 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libmanagement.
52 kb 12 kb 0 kb
/usr/lib/jvm/java-6-sun-1.6.0.24/jre/lib/amd64/libverify.so
520 kb 8 kb 0 kb /lib/libm-2.11.1.so
8 kb 8 kb 0 kb /lib/libdl-2.11.1.so
16 kb 0 kb 8 kb
4 kb 0 kb 4 kb /usr/bin/jsvc
438656 kb 0 kb 4 kb [heap]
28 kb 4 kb 0 kb /lib/librt-2.11.1.so
4 kb 0 kb 4 kb /lib/libnss_files-2.11.1.so
4 kb 0 kb 4 kb /lib/libnss_nis-2.11.1.so
4 kb 0 kb 4 kb /lib/libnsl-2.11.1.so
4 kb 0 kb 4 kb /lib/libnsl-2.11.1.so
4 kb 0 kb 4 kb /lib/libnss_compat-2.11.1.so
4 kb 0 kb 4 kb /lib/libpthread-2.11.1.so
4 kb 0 kb 4 kb /lib/libdl-2.11.1.so
16 kb 0 kb 4 kb
4 kb 0 kb 4 kb
4 kb 4 kb 0 kb [vdso]
