It was in AWS that I had clocks by off by 30 seconds or so. Virtualization is a nightmare for clocks.
As long as you've checked, we can move onto other possibilities :) > On Nov 13, 2015, at 1:28 PM, Peddi, Praveen <pe...@amazon.com> wrote: > > Lol… > We are running on AWS servers and no clocks are not 20 minutes off. > > > From: Jon Haddad <jonathan.had...@gmail.com > <mailto:jonathan.had...@gmail.com>> > Reply-To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org <mailto:user@cassandra.apache.org>> > Date: Friday, November 13, 2015 at 4:24 PM > To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>" > <user@cassandra.apache.org <mailto:user@cassandra.apache.org>> > Subject: Re: Deletes Reappeared even when nodes are not down > > Well, yeah, it's still possible. Are your clocks 20 minutes off? > >> On Nov 13, 2015, at 1:20 PM, Peddi, Praveen <pe...@amazon.com >> <mailto:pe...@amazon.com>> wrote: >> >> Hi Jon, >> Thanks for your response. >> Clock skews is not a possibility because the row that got re-appeared is 20 >> mins older than the one got deleted (based on last modified date field). We >> are definitely not talking about few millis here. >> >> Praveen >> >> From: Jon Haddad <jonathan.had...@gmail.com >> <mailto:jonathan.had...@gmail.com>> >> Reply-To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>" >> <user@cassandra.apache.org <mailto:user@cassandra.apache.org>> >> Date: Friday, November 13, 2015 at 4:13 PM >> To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>" >> <user@cassandra.apache.org <mailto:user@cassandra.apache.org>> >> Subject: Re: Deletes Reappeared even when nodes are not down >> >> Any chance your clocks are off? >> >>> On Nov 13, 2015, at 1:09 PM, Peddi, Praveen <pe...@amazon.com >>> <mailto:pe...@amazon.com>> wrote: >>> >>> Hi, >>> We are using Cassandra 2.0.8, with replication factor of 3. >>> >>> We are seeing a scenario where some of the rows in the table reappears even >>> after they are deleted. We have seen this in Prod 3 times in last 1 week >>> and coincidentally all 3 times on the same partition. We have confirmed >>> that nodes didn’t go down (they were all started on Oct 28). I do not see >>> any resource issues (CPU is around 5%, memory < 10%, iostats looks normal >>> etc) on any nodes. >>> >>> The only possible explanation we have so far is, one node that has data for >>> affected partition, being browned out for at least 15 minutes (our GC grace >>> is 15 minutes) and came back up but when it came back up, there were some >>> rows in that partition that were deleted on other two nodes. However I am >>> not sure how to prove this is what is happening. >>> >>> I am not sure if this is a bug in Cassandra (unlikely since we have not >>> seen this issue in last 8 months until this week). Any pointers as to how >>> to find the root cause would be greatly appreciated. >>> >>> Thanks in advance. >>> Praveen >> >
