Hello Patrick, Still in trouble with this? I must admit I'm really puzzled by your issue. I have no real idea of what's going on. Would you share with us the output of:
- nodetool status <keyspace> - nodetool describecluster - nodetool gossipinfo - nodetool tpstats Also you said the app is running for a long time, with no changes. What about Cassandra? Any recent operations? I hope that with this information we might be able to understand better and finally be able to help. ----------------------- Alain Rodriguez - al...@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com Le ven. 4 oct. 2019 à 00:25, Patrick Lee <patrickclee0...@gmail.com> a écrit : > this table was actually leveled compaction before, just changed it to size > tiered yesterday while researching this. > > On Thu, Oct 3, 2019 at 4:31 PM Patrick Lee <patrickclee0...@gmail.com> > wrote: > >> its not really time series data. and it's not updated very often, it >> would have some updates but pretty infrequent. this thing should be super >> fast, on avg it's like 1 to 2ms p99 currently but if they double - triple >> the traffic on that table latencies go upward to 20ms to 50ms.. the only >> odd thing i see is just that there are constant read repairs that follow >> the same traffic pattern on the reads, which shows constant writes on the >> table (from the read repairs), which after read repair or just normal full >> repairs (all full through reaper, never ran any incremental repair) i would >> expect it to not have any mismatches. the other 5 tables they use on the >> cluster can have the same level traffic all very simple select from table >> by partition key which returns a single record >> >> On Thu, Oct 3, 2019 at 4:21 PM John Belliveau <belliveau.j...@gmail.com> >> wrote: >> >>> Hi Patrick, >>> >>> >>> >>> Is this time series data? If so, I have run into issues with repair on >>> time series data using the SizeTieredCompactionStrategy. I have had >>> better luck using the TimeWindowCompactionStrategy. >>> >>> >>> >>> John >>> >>> >>> >>> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for >>> Windows 10 >>> >>> >>> >>> *From: *Patrick Lee <patrickclee0...@gmail.com> >>> *Sent: *Thursday, October 3, 2019 5:14 PM >>> *To: *user@cassandra.apache.org >>> *Subject: *Constant blocking read repair for such a tiny table >>> >>> >>> >>> I have a cluster that is running 3.11.4 ( was upgraded a while back from >>> 2.1.16 ). what I see is a steady rate of read repair which is about 10% >>> constantly on only this 1 table. Repairs have been run (actually several >>> times). The table does not have a lot of writes to it so after repair, or >>> even after a read repair I would expect it to be fine. the reason i'm >>> having to dig into this so much is for the fact that under a much large >>> traffic load than their normal traffic, latencies are higher than the app >>> team wants >>> >>> >>> >>> I mean this thing is tiny, it's a 12x12 cluster but this 1 table is like >>> 1GB per node on disk. >>> >>> >>> >>> the application team is doing reads at LOCAL_QUORUM and I can simulate >>> this on that cluster by running a query using quorum and/or local_quorum >>> and in the trace can see every time running the query it comes back with a >>> DigestMismatchException no matter how many times I run it. that record >>> hasn't been updated by the application for several months. >>> >>> >>> >>> repairs are scheduled and run every 7 days via reaper, recently in the >>> past week this table has been repaired at least 3 times. every time there >>> are mismatches and data streams back and forth but yet still a constant >>> rate of read repairs. >>> >>> >>> >>> curious if anyone has any recommendations to look info further or have >>> experienced anything like this? >>> >>> >>> >>> this node has been up for 24 hours.. this is the netstats for read >>> repairs >>> >>> Mode: NORMAL >>> Not sending any streams. >>> Read Repair Statistics: >>> Attempted: 7481 >>> Mismatch (Blocking): 11425375 >>> Mismatch (Background): 17 >>> Pool Name Active Pending Completed Dropped >>> Large messages n/a 0 1232 0 >>> Small messages n/a 0 395903678 0 >>> Gossip messages n/a 0 603746 0 >>> >>> >>> >>> example of the schema... some modifications have been made to reduce >>> read_reapair and speculative_retry while troubleshooting.. >>> >>> CREATE TABLE keyspace.table1 ( >>> >>> item bigint, >>> >>> price int, >>> >>> start_date timestamp, >>> >>> end_date timestamp, >>> >>> created_date timestamp, >>> >>> cost decimal, >>> >>> list decimal, >>> >>> item_id int, >>> >>> modified_date timestamp, >>> >>> status int, >>> >>> PRIMARY KEY ((item, price), start_date, end_date) >>> >>> ) WITH CLUSTERING ORDER BY (start_date ASC, end_date ASC) >>> >>> AND read_repair_chance = 0.0 >>> >>> AND dclocal_read_repair_chance = 0.0 >>> >>> AND gc_grace_seconds = 864000 >>> >>> AND bloom_filter_fp_chance = 0.01 >>> >>> AND caching = { 'keys' : 'ALL', 'rows_per_partition' : 'NONE' } >>> >>> AND comment = '' >>> >>> AND compaction = { 'class' : >>> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', >>> 'max_threshold' : 32, 'min_threshold' : 4 } >>> >>> AND compression = { 'chunk_length_in_kb' : 4, 'class' : >>> 'org.apache.cassandra.io.compress.LZ4Compressor' } >>> >>> AND default_time_to_live = 0 >>> >>> AND speculative_retry = 'NONE' >>> >>> AND min_index_interval = 128 >>> >>> AND max_index_interval = 2048 >>> >>> AND crc_check_chance = 1.0 >>> >>> AND cdc = false >>> >>> AND memtable_flush_period_in_ms = 0; >>> >>> >>> >>
