Re: Deleted columns reappear after "repair"

2015-03-26 Thread Roman Tkachenko
Yep, good point: https://issues.apache.org/jira/browse/CASSANDRA-9045. On Thu, Mar 26, 2015 at 4:23 PM, Robert Coli wrote: > On Wed, Mar 25, 2015 at 6:53 PM, Roman Tkachenko > wrote: > >> Yup, I increased "in_memory_compaction_limit_in_mb" to 512MB so the row >> in question fits into it and ran

Re: Deleted columns reappear after "repair"

2015-03-26 Thread Robert Coli
On Wed, Mar 25, 2015 at 6:53 PM, Roman Tkachenko wrote: > Yup, I increased "in_memory_compaction_limit_in_mb" to 512MB so the row in > question fits into it and ran repair on a couple of nodes owning its key. > The log entries about this particular row went away and those columns > haven't reappe

Re: Deleted columns reappear after "repair"

2015-03-25 Thread Roman Tkachenko
Thanks Robert. Yup, I increased "in_memory_compaction_limit_in_mb" to 512MB so the row in question fits into it and ran repair on a couple of nodes owning its key. The log entries about this particular row went away and those columns haven't reappeared, yet. If that was the reason, that's unfortun

Re: Deleted columns reappear after "repair"

2015-03-25 Thread Robert Coli
On Wed, Mar 25, 2015 at 1:57 PM, Roman Tkachenko wrote: > Okay, so I'm positively going crazy :) > > Increasing gc_grace + repair + decreasing gc_grace didn't help. The > columns still appear after the repair. I checked in cassandra-cli and > timestamps for these columns are old, not in the futur

Re: Deleted columns reappear after "repair"

2015-03-25 Thread Roman Tkachenko
Okay, so I'm positively going crazy :) Increasing gc_grace + repair + decreasing gc_grace didn't help. The columns still appear after the repair. I checked in cassandra-cli and timestamps for these columns are old, not in the future, so it shouldn't be the reason. I also did a test: updated one o

Re: Deleted columns reappear after "repair"

2015-03-24 Thread Roman Tkachenko
Well, as I mentioned in my original email all machines running Cassandra are running NTP. This was one of the first things I verified and I triple checked that they all show the same time. Is this sufficient to ensure clocks are synched between the nodes? I have increased gc_grace to 100 days for

Re: Deleted columns reappear after "repair"

2015-03-24 Thread Duncan Sands
Hi Roman, On 24/03/15 18:05, Roman Tkachenko wrote: Hi Duncan, Thanks for the response! I can try increasing gc_grace_seconds and run repair on all nodes. It does not make sense though why all *new* deletes (for the same column that resurrects after repair) I do are forgotten as well after rep

Re: Deleted columns reappear after "repair"

2015-03-24 Thread Roman Tkachenko
Hi Duncan, Thanks for the response! I can try increasing gc_grace_seconds and run repair on all nodes. It does not make sense though why all *new* deletes (for the same column that resurrects after repair) I do are forgotten as well after repair? Doesn't Cassandra insert a new tombstone every tim

Re: Deleted columns reappear after "repair"

2015-03-24 Thread Duncan Sands
Hi Roman, On 24/03/15 17:32, Roman Tkachenko wrote: Hey guys, Has anyone seen anything like this behavior or has an explanation for it? If not, I think I'm gonna file a bug report. this can happen if repair is run after the tombstone gc_grace_period has expired. I suggest you increase gc_gr

Re: Deleted columns reappear after "repair"

2015-03-24 Thread Roman Tkachenko
Hey guys, Has anyone seen anything like this behavior or has an explanation for it? If not, I think I'm gonna file a bug report. Thanks! Roman On Mon, Mar 23, 2015 at 4:45 PM, Roman Tkachenko wrote: > Hey guys, > > We're having a very strange issue: deleted columns get resurrected when > "rep

Deleted columns reappear after "repair"

2015-03-23 Thread Roman Tkachenko
Hey guys, We're having a very strange issue: deleted columns get resurrected when "repair" is run on a node. Info about the setup. Cassandra 2.0.13, multi datacenter with 12 nodes in one datacenter and 6 nodes in another one. Schema: cqlsh> describe keyspace blackbook; CREATE KEYSPACE blackbook