there is a setting in the yaml file that helps relieve memory pressure by reducing the row cache. it is based on the percent of memory used by the JVM
the setting are, reduce_cache_sizes_at and reduce_cache_capacity_to. see how much free memory you have and if the numbers suggest that you have hit that limit, therefore reducing row cache size From: Eran Chinthaka Withana <eran.chinth...@gmail.com<mailto:eran.chinth...@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Thu, 16 Feb 2012 13:52:38 -0800 To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Key cache hit rate issue Hi Jonathan, Thanks for the reply. Yes there is a possibility that the keys can be distributed in multiple SSTables, but my data access patterns are such that I always read/write the whole row. So I expect all the data to be in the same SSTable (please correct me if I'm wrong). For some reason 16637958 (the keys cached) has become a golden number and I don't see key cache increasing beyond that. I also checked memory and I have about 4GB left in JVM memory and didn't see any issues on logs. Thanks, Eran Chinthaka Withana On Thu, Feb 16, 2012 at 1:20 PM, Jonathan Ellis <jbel...@gmail.com<mailto:jbel...@gmail.com>> wrote: So, you have roughly 1/6 of your (physical) row keys cached and about 1/4 cache hit rate, which doesn't sound unreasonable to me. Remember, each logical key may be spread across multiple physical sstables -- each (key, sstable) pair is one entry in the key cache. On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana <eran.chinth...@gmail.com<mailto:eran.chinth...@gmail.com>> wrote: > Hi Aaron, > > Here it is. > > Keyspace: XXXX > Read Count: 1123637972 > Read Latency: 5.757938114343114 ms. > Write Count: 128201833 > Write Latency: 0.0682576607387509 ms. > Pending Tasks: 0 > Column Family: YY > SSTable count: 18 > Space used (live): 103318720685 > Space used (total): 103318720685 > Number of Keys (estimate): 92404992 > Memtable Columns Count: 1425580 > Memtable Data Size: 359655747 > Memtable Switch Count: 2522 > Read Count: 1123637972 > Read Latency: 14.731 ms. > Write Count: 128201833 > Write Latency: NaN ms. > Pending Tasks: 0 > Bloom Filter False Postives: 1488 > Bloom Filter False Ratio: 0.00000 > Bloom Filter Space Used: 331522920 > Key cache capacity: 16637958 > Key cache size: 16637958 > Key cache hit rate: 0.2708333333333333 > Row cache: disabled > Compacted row minimum size: 51 > Compacted row maximum size: 6866 > Compacted row mean size: 2560 > > Thanks, > Eran Chinthaka Withana > > > > On Thu, Feb 16, 2012 at 12:30 AM, aaron morton > <aa...@thelastpickle.com<mailto:aa...@thelastpickle.com>> > wrote: >> >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is >> bit high. Is there a way to fix/improve it? >> >> Sorry I don't understand what you mean. But if the ratio is 0.0 all is >> good. >> >> Could you include the full output from cfstats for the CF you are looking >> at ? >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote: >> >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is >> bit high. Is there a way to fix/improve it? >> >> Thanks, >> Eran Chinthaka Withana >> >> >> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton >> <aa...@thelastpickle.com<mailto:aa...@thelastpickle.com>> >> wrote: >>> >>> Out of interest what does cfstats say about the bloom filter stats ? A >>> high false positive could lead to a low key cache hit rate. >>> >>> Also, is there a way to warm start the key cache, meaning pre-load the >>> amount of keys I set as keys_cached? >>> >>> See key_cache_save_period when creating the CF. >>> >>> Cheers >>> >>> >>> ----------------- >>> Aaron Morton >>> Freelance Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>> >>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote: >>> >>> Hi, >>> >>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80% >>> (using the numerical values). This is visible in cfstats too. But I'm >>> getting less than 20% (or sometimes even 0%) key cache hit rate. Well, the >>> data access pattern is not the issue here as I know they are retrieving the >>> same row multiple times. I'm using hector client with dynamic load balancing >>> policy with consistency ONE for both reads and writes. Any ideas on how to >>> find the issue and fix this? >>> >>> Here is what I see on cfstats. >>> >>> Key cache capacity: 16637958 >>> Key cache size: 16637958 >>> Key cache hit rate: 0.045454545454545456 >>> >>> Also, is there a way to warm start the key cache, meaning pre-load the >>> amount of keys I set as keys_cached? >>> >>> Thanks, >>> Eran >>> >>> >> >> > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
