I never used JMX for any changes and use JMX only for monitoring. All my
updates goes through schema updates.
To give you little bit more context (not sure whether this will help but
anyway), about 2-3 weeks back the read latency was 4-8ms with about 90-95%
key cache hit rate. But after that point, I stopped all the reads and kept
only writes into the system. When I enabled reads this week, then only I
saw this read latency degradation. I didn't run nodetool repair for
sometime (gc grace is 10 days), since I stopped readers. Can this be
connected to the read degradation? I started running repairs last night but
didn't see any read latency improvements, yet.
Thanks,
Eran Chinthaka Withana
On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis <jbel...@gmail.com> wrote:
> Only thing I can think of is that if you've set the cache size
> manually over JMX it will preserve that size if you change it via a
> schema update.
>
> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
> <eran.chinth...@gmail.com> wrote:
> > Hi Jonathan,
> >>
> >>
> >> > For some reason 16637958 (the keys cached) has become a golden number
> >> > and I
> >> > don't see key cache increasing beyond that.
> >>
> >> 16637958 is your configured cache capacity according to the cfstats you
> >> pasted.
> >
> >
> > this is another weird part. If you look at the schema[1] (pasted here,
> > extracted through cli), the keys_cached is set to 40million. And cfstats
> > shows that cassandra is only using 16 million.
> >
> > Please note that I enabled row cache since I posted the cfstats. And
> pasting
> > the latest cfstats[2] herewith.
> >
> > [1]
> > create column family YY
> > with column_type = 'Standard'
> > and comparator = 'UTF8Type'
> > and default_validation_class = 'UTF8Type'
> > and key_validation_class = 'LongType'
> > and rows_cached = 10000.0
> > and row_cache_save_period = 0
> > and row_cache_keys_to_save = 2147483647
> > and keys_cached = 4.0E7
> > and key_cache_save_period = 14400
> > and read_repair_chance = 0.1
> > and gc_grace = 864000
> > and min_compaction_threshold = 4
> > and max_compaction_threshold = 32
> > and replicate_on_write = true
> > and row_cache_provider = 'SerializingCacheProvider'
> > and compaction_strategy =
> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> > and compression_options = {'chunk_length_kb' : '64',
> 'sstable_compression'
> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> >
> > [2]
> > Keyspace: XXXXX
> > Read Count: 590667
> > Read Latency: 11.644107261790484 ms.
> > Write Count: 12720068
> > Write Latency: 0.06049896832312532 ms.
> > Pending Tasks: 0
> > Column Family: YY
> > SSTable count: 12
> > Space used (live): 66300297338
> > Space used (total): 66300297338
> > Number of Keys (estimate): 61152896
> > Memtable Columns Count: 881972
> > Memtable Data Size: 211749842
> > Memtable Switch Count: 267
> > Read Count: 590667
> > Read Latency: 16.859 ms.
> > Write Count: 12720068
> > Write Latency: 0.060 ms.
> > Pending Tasks: 0
> > Bloom Filter False Postives: 507
> > Bloom Filter False Ratio: 0.00099
> > Bloom Filter Space Used: 168118968
> > Key cache capacity: 40000000
> > Key cache size: 3838755
> > Key cache hit rate: 0.2072072072072072
> > Row cache capacity: 10000
> > Row cache size: 10000
> > Row cache hit rate: 0.022222222222222223
> > Compacted row minimum size: 51
> > Compacted row maximum size: 6866
> > Compacted row mean size: 2577
> >
> > Thanks,
> > Eran
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
