Well, yeah, it's still possible. Are your clocks 20 minutes off?
> On Nov 13, 2015, at 1:20 PM, Peddi, Praveen <pe...@amazon.com> wrote:
>
> Hi Jon,
> Thanks for your response.
> Clock skews is not a possibility because the row that got re-appeared is 20
> mins older than the one got deleted (based on last modified date field). We
> are definitely not talking about few millis here.
>
> Praveen
>
> From: Jon Haddad <jonathan.had...@gmail.com
> <mailto:jonathan.had...@gmail.com>>
> Reply-To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>"
> <user@cassandra.apache.org <mailto:user@cassandra.apache.org>>
> Date: Friday, November 13, 2015 at 4:13 PM
> To: "user@cassandra.apache.org <mailto:user@cassandra.apache.org>"
> <user@cassandra.apache.org <mailto:user@cassandra.apache.org>>
> Subject: Re: Deletes Reappeared even when nodes are not down
>
> Any chance your clocks are off?
>
>> On Nov 13, 2015, at 1:09 PM, Peddi, Praveen <pe...@amazon.com
>> <mailto:pe...@amazon.com>> wrote:
>>
>> Hi,
>> We are using Cassandra 2.0.8, with replication factor of 3.
>>
>> We are seeing a scenario where some of the rows in the table reappears even
>> after they are deleted. We have seen this in Prod 3 times in last 1 week and
>> coincidentally all 3 times on the same partition. We have confirmed that
>> nodes didn’t go down (they were all started on Oct 28). I do not see any
>> resource issues (CPU is around 5%, memory < 10%, iostats looks normal etc)
>> on any nodes.
>>
>> The only possible explanation we have so far is, one node that has data for
>> affected partition, being browned out for at least 15 minutes (our GC grace
>> is 15 minutes) and came back up but when it came back up, there were some
>> rows in that partition that were deleted on other two nodes. However I am
>> not sure how to prove this is what is happening.
>>
>> I am not sure if this is a bug in Cassandra (unlikely since we have not seen
>> this issue in last 8 months until this week). Any pointers as to how to find
>> the root cause would be greatly appreciated.
>>
>> Thanks in advance.
>> Praveen
>
