Well, as I mentioned in my original email all machines running Cassandra are running NTP. This was one of the first things I verified and I triple checked that they all show the same time. Is this sufficient to ensure clocks are synched between the nodes?
I have increased gc_grace to 100 days for now and am running repair on the affected keyspace, it should be done today. In the meanwhile if you (or anyone else) have other ideas / suggestions on how to debug this, they're much appreciated. Thanks for your help! Roman On Tue, Mar 24, 2015 at 10:39 AM, Duncan Sands <duncan.sa...@gmail.com> wrote: > Hi Roman, > > On 24/03/15 18:05, Roman Tkachenko wrote: > >> Hi Duncan, >> >> Thanks for the response! >> >> I can try increasing gc_grace_seconds and run repair on all nodes. It >> does not >> make sense though why all *new* deletes (for the same column that >> resurrects >> after repair) I do are forgotten as well after repair? Doesn't Cassandra >> insert >> a new tombstone every time delete happens? >> > > it does. Maybe the data you are trying to delete has a timestamp > (writetime) in the future, for example because clocks aren't synchronized > between your nodes. > > >> Also, how do I find out the value to set gc_grace_seconds to? >> > > It needs to be big enough that you are sure to repair your entire cluster > in less than that time. For example, observe how long repairing the entire > cluster takes and multiply by 3 or 4 (in case a repair fails or is > interrupted one day). > > Once incremental repair is solid maybe the whole gc_grace thing will > eventually go away, eg by modifying C* to only drop known repaired > tombstones. > > Ciao, Duncan. > > >> Thanks. >> >> On Tue, Mar 24, 2015 at 9:38 AM, Duncan Sands <duncan.sa...@gmail.com >> <mailto:duncan.sa...@gmail.com>> wrote: >> >> Hi Roman, >> >> On 24/03/15 17:32, Roman Tkachenko wrote: >> >> Hey guys, >> >> Has anyone seen anything like this behavior or has an explanation >> for it? If >> not, I think I'm gonna file a bug report. >> >> >> this can happen if repair is run after the tombstone gc_grace_period >> has >> expired. I suggest you increase gc_grace_period. >> >> Ciao, Duncan. >> >> >> >
