Hi Jon, Thanks for your response. Clock skews is not a possibility because the row that got re-appeared is 20 mins older than the one got deleted (based on last modified date field). We are definitely not talking about few millis here.
Praveen From: Jon Haddad <jonathan.had...@gmail.com<mailto:jonathan.had...@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Friday, November 13, 2015 at 4:13 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Deletes Reappeared even when nodes are not down Any chance your clocks are off? On Nov 13, 2015, at 1:09 PM, Peddi, Praveen <pe...@amazon.com<mailto:pe...@amazon.com>> wrote: Hi, We are using Cassandra 2.0.8, with replication factor of 3. We are seeing a scenario where some of the rows in the table reappears even after they are deleted. We have seen this in Prod 3 times in last 1 week and coincidentally all 3 times on the same partition. We have confirmed that nodes didn't go down (they were all started on Oct 28). I do not see any resource issues (CPU is around 5%, memory < 10%, iostats looks normal etc) on any nodes. The only possible explanation we have so far is, one node that has data for affected partition, being browned out for at least 15 minutes (our GC grace is 15 minutes) and came back up but when it came back up, there were some rows in that partition that were deleted on other two nodes. However I am not sure how to prove this is what is happening. I am not sure if this is a bug in Cassandra (unlikely since we have not seen this issue in last 8 months until this week). Any pointers as to how to find the root cause would be greatly appreciated. Thanks in advance. Praveen
