I have a situation where the off-heap memory is bloating the jvm process
memory, making it a candidate to be killed by the oom_killer.
My server has 256 GB RAM and Cassandra heap memory of 16 GB
Below is the output of "nodetool info" and nodetool compactionstats for a
culprit table which causes bloom filter bloat.
Ofcourse one option is to turnoff bloom filter, but I need to look into
application access pattern, etc.
xss = -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp
-javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M
-XX:+HeapDumpOnOutOfMemoryError -Xss256k
ID : 2b9b4252-0760-49c1-8d14-544be0183271
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 953.19 GB
Generation No : 1488641545
Uptime (seconds) : 15706
Heap Memory (MB) : 7692.93 / 16309.00
Off Heap Memory (MB) : 175115.07
Data Center : ord
Rack : rack3
Exceptions : 0
Key Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0
requests, NaN recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0
requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0
requests, NaN recent hit rate, 7200 save period in seconds
Token : (invoke with -T/--tokens to see all 256 tokens)
Table: logs_by_user
SSTable count: 622
SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 313156769247
Space used (total): 313156769247
Space used by snapshots (total): 0
Off heap memory used (total): 180354511884
SSTable Compression Ratio: 0.25016314078395613
Number of keys (estimate): 147261312
Memtable cell count: 44796
Memtable data size: 57578717
Memtable off heap memory used: 0
Memtable switch count: 21
Local read count: 0
Local read latency: NaN ms
Local write count: 1148687
Local write latency: 0.123 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 180269125192
Bloom filter off heap memory used: 180269120216
Index summary off heap memory used: 24335340
Compression metadata off heap memory used: 61056328
Compacted partition minimum bytes: 150
Compacted partition maximum bytes: 668489532
Compacted partition mean bytes: 3539
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
From: Conversant <jthak<mailto:jthak...@conversantmedia.com>
