Discussions previously on the list show why this is not a problem in much more detail.
If something changes in your cluster: node down, new node, etc - you run repair for sure. We also run periodic repairs prophylactically. But if you never delete and always ttl by the same amount, you do not have to worry about zombie data being resurrected - the main reason for running repair within gc_grace_seconds. On Tue, Apr 21, 2015 at 11:49 AM, Walsh, Stephen <stephen.wa...@aspect.com> wrote: > Maybe thanks Michael, > > I will give these setting a go, > > How do you do you periodic node-tool repairs in the situation, for what I > read we need to start doing this also. > > > > https://wiki.apache.org/cassandra/Operations#Frequency_of_nodetool_repair > > > > > > *From:* Laing, Michael [mailto:michael.la...@nytimes.com] > *Sent:* 21 April 2015 16:26 > *To:* user@cassandra.apache.org > *Subject:* Re: Cassandra tombstones being created by updating rows with > TTL's > > > > If you never delete except by ttl, and always write with the same ttl (or > monotonically increasing), you can set gc_grace_seconds to 0. > > > > That's what we do. There have been discussions on the list over the last > few years re this topic. > > > > ml > > > > On Tue, Apr 21, 2015 at 11:14 AM, Walsh, Stephen <stephen.wa...@aspect.com> > wrote: > > We were chatting to Jon Haddena about a week ago about our tombstone > issue using Cassandra 2.0.14 > > To Summarize > > > > We have a 3 node cluster with replication-factor=3 and compaction = > SizeTiered > > We use 1 keyspace with 1 table > > Each row have about 40 columns > > Each row has a TTL of 10 seconds > > > > We insert about 500 rows per second in a prepared batch** (about 3mb in > network overhead) > > We query the entire table once per second > > > > **This is too enable consistent data, E.G batch in transactional, so we > get all queried data from one insert and not a mix of 2 or more. > > > > > > Seems every second we insert, the rows are never deleted by the TTL, or so > we thought. > > After some time we got this message on the query side > > > > > > ####################################### > > ERROR [ReadStage:91] 2015-04-21 12:27:03,902 SliceQueryFilter.java (line > 206) Scanned over 100000 tombstones in keyspace.table; query aborted (see > tombstone_failure_threshold) > > ERROR [ReadStage:91] 2015-04-21 12:27:03,931 CassandraDaemon.java (line > 199) Exception in thread Thread[ReadStage:91,5,main] > > java.lang.RuntimeException: > org.apache.cassandra.db.filter.TombstoneOverwhelmingException > > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2008) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: org.apache.cassandra.db.filter.TombstoneOverwhelmingException > > ####################################### > > > > > > So we know tombstones are infact being created. > > Solution was to change the table schema and set gc_grace_seconds to run > every 60 seconds. > > This worked for 20 seconds, then we saw this > > > > > > ####################################### > > Read 500 live and 30000 tombstoned cells in keyspace.table (see > tombstone_warn_threshold). 10000 columns was requested, slices=[-], > delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647} > > ####################################### > > > > So every 20 seconds (500 inserts x 20 seconds = 10,000 tombstones) > > So now we have the gc_grace_seconds set to 10 seoncds. > > But its feels very wrong to have it at a low number, especially if we move > to a larger cluster. This just wont fly. > > What are we doing wrong? > > > > We shouldn’t increase the tombstone threshold as that is extremely > dangerous. > > > > > > Best Regards > > Stephen Walsh > > > > > > > > > > > > > > This email (including any attachments) is proprietary to Aspect Software, > Inc. and may contain information that is confidential. If you have received > this message in error, please do not read, copy or forward this message. > Please notify the sender immediately, delete it from your system and > destroy any copies. You may not further disclose or distribute this email > or its attachments. > > > This email (including any attachments) is proprietary to Aspect > Software, Inc. and may contain information that is confidential. If you > have received this message in error, please do not read, copy or forward > this message. Please notify the sender immediately, delete it from your > system and destroy any copies. You may not further disclose or distribute > this email or its attachments. >
