Thanks Jeff, I'm having gcgs set to 10 mins and changed the table ttl also to 5 hours compared to insert ttl to 4 hours . Tracing on doesn't show any tombstone scans for the reads. And also log doesn't show tombstone scan alerts. Has the reads are happening 5-8k reads per node during the peak hours it shows 1M tombstone scans count per read.
On Fri, Feb 22, 2019, 11:46 AM Jeff Jirsa <jji...@gmail.com> wrote: > If all of your data is TTL’d and you never explicitly delete a cell > without using s TTL, you can probably drop your GCGS to 1 hour (or less). > > Which compaction strategy are you using? You need a way to clear out those > tombstones. There exist tombstone compaction sub properties that can help > encourage compaction to grab sstables just because they’re full of > tombstones which will probably help you. > > > -- > Jeff Jirsa > > > On Feb 22, 2019, at 8:37 AM, Kenneth Brotman <kenbrot...@yahoo.com.invalid> > wrote: > > Can we see the histogram? Why wouldn’t you at times have that many > tombstones? Makes sense. > > > > Kenneth Brotman > > > > *From:* Rahul Reddy [mailto:rahulreddy1...@gmail.com > <rahulreddy1...@gmail.com>] > *Sent:* Thursday, February 21, 2019 7:06 AM > *To:* user@cassandra.apache.org > *Subject:* Tombstones in memtable > > > > We have small table records are about 5k . > > All the inserts comes as 4hr ttl and we have table level ttl 1 day and gc > grace seconds has 3 hours. We do 5k reads a second during peak load During > the peak load seeing Alerts for tomstone scanned histogram reaching million. > > Cassandra version 3.11.1. Please let me know how can this tombstone scan > can be avoided in memtable > >
