Thanks Hannu - also considered that option. However, that's a trial and error and will have to play with the collision/false-positive fraction. And each iteration will most likely result in a compaction storm - so I was hoping for a way to throttle/limit the max off-heap size.
The reason I was thinking of eliminating bloom filters is because due to application design, we search for data using a partial key (prefix columns), hence am thinking of completely eliminating the bloom filters as they do not add any value in such a use case. Is my assumption correct? From: Hannu Kröger <hkro...@gmail.com> Date: Sunday, March 5, 2017 at 6:34 AM To: <user@cassandra.apache.org> Subject: Re: Any way to control/limit off-heap memory? If bloom filters are taking too much memory, you can adjust bloom filters: https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html Hannu On 4 Mar 2017, at 22:54, Thakrar, Jayesh <jthak...@conversantmedia.com<mailto:jthak...@conversantmedia.com>> wrote: I have a situation where the off-heap memory is bloating the jvm process memory, making it a candidate to be killed by the oom_killer. My server has 256 GB RAM and Cassandra heap memory of 16 GB Below is the output of "nodetool info" and nodetool compactionstats for a culprit table which causes bloom filter bloat. Ofcourse one option is to turnoff bloom filter, but I need to look into application access pattern, etc. xss = -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k ID : 2b9b4252-0760-49c1-8d14-544be0183271 Gossip active : true Thrift active : false Native Transport active: true Load : 953.19 GB Generation No : 1488641545 Uptime (seconds) : 15706 Heap Memory (MB) : 7692.93 / 16309.00 Off Heap Memory (MB) : 175115.07 Data Center : ord Rack : rack3 Exceptions : 0 Key Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 14400 save period in seconds Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds Counter Cache : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds Token : (invoke with -T/--tokens to see all 256 tokens) Table: logs_by_user SSTable count: 622 SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0] Space used (live): 313156769247 Space used (total): 313156769247 Space used by snapshots (total): 0 Off heap memory used (total): 180354511884 SSTable Compression Ratio: 0.25016314078395613 Number of keys (estimate): 147261312 Memtable cell count: 44796 Memtable data size: 57578717 Memtable off heap memory used: 0 Memtable switch count: 21 Local read count: 0 Local read latency: NaN ms Local write count: 1148687 Local write latency: 0.123 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used: 180269125192 Bloom filter off heap memory used: 180269120216 Index summary off heap memory used: 24335340 Compression metadata off heap memory used: 61056328 Compacted partition minimum bytes: 150 Compacted partition maximum bytes: 668489532 Compacted partition mean bytes: 3539 Average live cells per slice (last five minutes): NaN Maximum live cells per slice (last five minutes): 0 Average tombstones per slice (last five minutes): NaN Maximum tombstones per slice (last five minutes): 0 From: Conversant <jthak<mailto:jthak...@conversantmedia.com>
