True, the high hit rate has translated to low read latency. But the
question is how can I debug the reason for low hit rate now assuming read
patterns haven't changed.
Thanks,
Eran Chinthaka Withana
On Fri, Feb 17, 2012 at 3:07 PM, Jonathan Ellis <jbel...@gmail.com> wrote:
> I suspect the main difference is that 2-3 weeks ago almost none of
> your reads had to hit disk.
>
> On Fri, Feb 17, 2012 at 1:53 PM, Eran Chinthaka Withana
> <eran.chinth...@gmail.com> wrote:
> > I never used JMX for any changes and use JMX only for monitoring. All my
> > updates goes through schema updates.
> >
> > To give you little bit more context (not sure whether this will help but
> > anyway), about 2-3 weeks back the read latency was 4-8ms with about
> 90-95%
> > key cache hit rate. But after that point, I stopped all the reads and
> kept
> > only writes into the system. When I enabled reads this week, then only I
> saw
> > this read latency degradation. I didn't run nodetool repair for sometime
> (gc
> > grace is 10 days), since I stopped readers. Can this be connected to the
> > read degradation? I started running repairs last night but didn't see any
> > read latency improvements, yet.
> >
> > Thanks,
> > Eran Chinthaka Withana
> >
> >
> >
> > On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis <jbel...@gmail.com>
> wrote:
> >>
> >> Only thing I can think of is that if you've set the cache size
> >> manually over JMX it will preserve that size if you change it via a
> >> schema update.
> >>
> >> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
> >> <eran.chinth...@gmail.com> wrote:
> >> > Hi Jonathan,
> >> >>
> >> >>
> >> >> > For some reason 16637958 (the keys cached) has become a golden
> number
> >> >> > and I
> >> >> > don't see key cache increasing beyond that.
> >> >>
> >> >> 16637958 is your configured cache capacity according to the cfstats
> you
> >> >> pasted.
> >> >
> >> >
> >> > this is another weird part. If you look at the schema[1] (pasted here,
> >> > extracted through cli), the keys_cached is set to 40million. And
> cfstats
> >> > shows that cassandra is only using 16 million.
> >> >
> >> > Please note that I enabled row cache since I posted the cfstats. And
> >> > pasting
> >> > the latest cfstats[2] herewith.
> >> >
> >> > [1]
> >> > create column family YY
> >> > with column_type = 'Standard'
> >> > and comparator = 'UTF8Type'
> >> > and default_validation_class = 'UTF8Type'
> >> > and key_validation_class = 'LongType'
> >> > and rows_cached = 10000.0
> >> > and row_cache_save_period = 0
> >> > and row_cache_keys_to_save = 2147483647
> >> > and keys_cached = 4.0E7
> >> > and key_cache_save_period = 14400
> >> > and read_repair_chance = 0.1
> >> > and gc_grace = 864000
> >> > and min_compaction_threshold = 4
> >> > and max_compaction_threshold = 32
> >> > and replicate_on_write = true
> >> > and row_cache_provider = 'SerializingCacheProvider'
> >> > and compaction_strategy =
> >> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> >> > and compression_options = {'chunk_length_kb' : '64',
> >> > 'sstable_compression'
> >> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> >> >
> >> > [2]
> >> > Keyspace: XXXXX
> >> > Read Count: 590667
> >> > Read Latency: 11.644107261790484 ms.
> >> > Write Count: 12720068
> >> > Write Latency: 0.06049896832312532 ms.
> >> > Pending Tasks: 0
> >> > Column Family: YY
> >> > SSTable count: 12
> >> > Space used (live): 66300297338
> >> > Space used (total): 66300297338
> >> > Number of Keys (estimate): 61152896
> >> > Memtable Columns Count: 881972
> >> > Memtable Data Size: 211749842
> >> > Memtable Switch Count: 267
> >> > Read Count: 590667
> >> > Read Latency: 16.859 ms.
> >> > Write Count: 12720068
> >> > Write Latency: 0.060 ms.
> >> > Pending Tasks: 0
> >> > Bloom Filter False Postives: 507
> >> > Bloom Filter False Ratio: 0.00099
> >> > Bloom Filter Space Used: 168118968
> >> > Key cache capacity: 40000000
> >> > Key cache size: 3838755
> >> > Key cache hit rate: 0.2072072072072072
> >> > Row cache capacity: 10000
> >> > Row cache size: 10000
> >> > Row cache hit rate: 0.022222222222222223
> >> > Compacted row minimum size: 51
> >> > Compacted row maximum size: 6866
> >> > Compacted row mean size: 2577
> >> >
> >> > Thanks,
> >> > Eran
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
