Hi all,
I am experiencing VERY poor key cache hit rate on my 6 node C* 1.2.4 with
Vnode cluster. I am using CQL3 with LCS and yesterday increased my SSTable
size from default 5 MB to 256 MB although I did not force a major compaction
and am instead letting the new size take effect organically as compactions are
normally triggered. The issue for this email is that my key cache hit rate is
TERRIBLE at < 1% and my bloom filter false positive rate is around 0.2 (see
cfstats , info, and create for example table). I am using the OOTB bloom
filter settings and only 300 Mbs of key cache. I'm guessing that I need to
increase my bloom_filter_fp_chance from default 0.1 to 0.01 (trigger
upgradesstables for it to take effect) and increase my key cache size (my nodes
have 32 GB of RAM and I'm running an 8 GB heap on java 7) but I thought I would
get everyone's opinion first. My read latency is good but I'm afraid I'll max
out the TPS on my SSD drives which are currently hovering around 40%
utilization.
Should I wait to see what effect the sstable size change has first as it will
obviously decrease the number of sstables?
What do you recommend for bloom filter and key cache values?
I tried upping the key cache size to 600 MB but did not see a sizable
difference in hit rate. I am also planning on trying out row cache but wanted
to hold off until I got these values handled first. FYI – each node has 3 x
800 GB SSD in JBOD running RF 3.
Thanks!
Column Family: cookie_user_lookup
SSTable count: 13464
SSTables in each level: [1, 3, 101/100, 1022/1000, 10587/10000, 1750]
Space used (live): 106151372078
Space used (total): 106870515200
Number of Keys (estimate): 954198528
Memtable Columns Count: 369605
Memtable Data Size: 170078697
Memtable Switch Count: 668
Read Count: 754323272
Read Latency: 1.029 ms.
Write Count: 130312827
Write Latency: 0.023 ms.
Pending Tasks: 0
Bloom Filter False Positives: 22785986
Bloom Filter False Ratio: 0.24048
Bloom Filter Space Used: 659403424
Compacted row minimum size: 36
Compacted row maximum size: 215
Compacted row mean size: 162
Token : (invoke with -T/--tokens to see all 256 tokens)
ID : 6e1a0ac1-6354-40e1-bba4-063c7bb0983d
Gossip active : true
Thrift active : true
Load : 204.64 GB
Generation No : 1374259760
Uptime (seconds) : 634791
Heap Memory (MB) : 5601.37 / 8072.00
Data Center : DAL1
Rack : RAC1
Exceptions : 78
Key Cache : size 314572552 (bytes), capacity 314572800 (bytes),
327775491 hits, 3259949221 requests, 0.162 recent hit rate, 14400 save period
in seconds
Row Cache : size 0 (bytes), capacity 0 (bytes), 0 hits, 0 requests, NaN
recent hit rate, 0 save period in seconds
CREATE TABLE cookie_user_lookup (
cookie text PRIMARY KEY,
creation_time timestamp,
opt_out boolean,
status int,
user_id timeuuid
) WITH
bloom_filter_fp_chance=0.100000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=86400 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'sstable_size_in_mb': '256', 'class':
'LeveledCompactionStrategy'} AND
compression={'chunk_length_kb': '8', 'crc_check_chance': '0.1',
'sstable_compression': 'LZ4Compressor'};
