I suspect the main difference is that 2-3 weeks ago almost none of
your reads had to hit disk.
On Fri, Feb 17, 2012 at 1:53 PM, Eran Chinthaka Withana
<eran.chinth...@gmail.com> wrote:
> I never used JMX for any changes and use JMX only for monitoring. All my
> updates goes through schema updates.
>
> To give you little bit more context (not sure whether this will help but
> anyway), about 2-3 weeks back the read latency was 4-8ms with about 90-95%
> key cache hit rate. But after that point, I stopped all the reads and kept
> only writes into the system. When I enabled reads this week, then only I saw
> this read latency degradation. I didn't run nodetool repair for sometime (gc
> grace is 10 days), since I stopped readers. Can this be connected to the
> read degradation? I started running repairs last night but didn't see any
> read latency improvements, yet.
>
> Thanks,
> Eran Chinthaka Withana
>
>
>
> On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis <jbel...@gmail.com> wrote:
>>
>> Only thing I can think of is that if you've set the cache size
>> manually over JMX it will preserve that size if you change it via a
>> schema update.
>>
>> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
>> <eran.chinth...@gmail.com> wrote:
>> > Hi Jonathan,
>> >>
>> >>
>> >> > For some reason 16637958 (the keys cached) has become a golden number
>> >> > and I
>> >> > don't see key cache increasing beyond that.
>> >>
>> >> 16637958 is your configured cache capacity according to the cfstats you
>> >> pasted.
>> >
>> >
>> > this is another weird part. If you look at the schema[1] (pasted here,
>> > extracted through cli), the keys_cached is set to 40million. And cfstats
>> > shows that cassandra is only using 16 million.
>> >
>> > Please note that I enabled row cache since I posted the cfstats. And
>> > pasting
>> > the latest cfstats[2] herewith.
>> >
>> > [1]
>> > create column family YY
>> > with column_type = 'Standard'
>> > and comparator = 'UTF8Type'
>> > and default_validation_class = 'UTF8Type'
>> > and key_validation_class = 'LongType'
>> > and rows_cached = 10000.0
>> > and row_cache_save_period = 0
>> > and row_cache_keys_to_save = 2147483647
>> > and keys_cached = 4.0E7
>> > and key_cache_save_period = 14400
>> > and read_repair_chance = 0.1
>> > and gc_grace = 864000
>> > and min_compaction_threshold = 4
>> > and max_compaction_threshold = 32
>> > and replicate_on_write = true
>> > and row_cache_provider = 'SerializingCacheProvider'
>> > and compaction_strategy =
>> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
>> > and compression_options = {'chunk_length_kb' : '64',
>> > 'sstable_compression'
>> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
>> >
>> > [2]
>> > Keyspace: XXXXX
>> > Read Count: 590667
>> > Read Latency: 11.644107261790484 ms.
>> > Write Count: 12720068
>> > Write Latency: 0.06049896832312532 ms.
>> > Pending Tasks: 0
>> > Column Family: YY
>> > SSTable count: 12
>> > Space used (live): 66300297338
>> > Space used (total): 66300297338
>> > Number of Keys (estimate): 61152896
>> > Memtable Columns Count: 881972
>> > Memtable Data Size: 211749842
>> > Memtable Switch Count: 267
>> > Read Count: 590667
>> > Read Latency: 16.859 ms.
>> > Write Count: 12720068
>> > Write Latency: 0.060 ms.
>> > Pending Tasks: 0
>> > Bloom Filter False Postives: 507
>> > Bloom Filter False Ratio: 0.00099
>> > Bloom Filter Space Used: 168118968
>> > Key cache capacity: 40000000
>> > Key cache size: 3838755
>> > Key cache hit rate: 0.2072072072072072
>> > Row cache capacity: 10000
>> > Row cache size: 10000
>> > Row cache hit rate: 0.022222222222222223
>> > Compacted row minimum size: 51
>> > Compacted row maximum size: 6866
>> > Compacted row mean size: 2577
>> >
>> > Thanks,
>> > Eran
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
