Any chance your clocks are off?
> On Nov 13, 2015, at 1:09 PM, Peddi, Praveen <pe...@amazon.com> wrote:
>
> Hi,
> We are using Cassandra 2.0.8, with replication factor of 3.
>
> We are seeing a scenario where some of the rows in the table reappears even
> after they are deleted. We have seen this in Prod 3 times in last 1 week and
> coincidentally all 3 times on the same partition. We have confirmed that
> nodes didn’t go down (they were all started on Oct 28). I do not see any
> resource issues (CPU is around 5%, memory < 10%, iostats looks normal etc) on
> any nodes.
>
> The only possible explanation we have so far is, one node that has data for
> affected partition, being browned out for at least 15 minutes (our GC grace
> is 15 minutes) and came back up but when it came back up, there were some
> rows in that partition that were deleted on other two nodes. However I am not
> sure how to prove this is what is happening.
>
> I am not sure if this is a bug in Cassandra (unlikely since we have not seen
> this issue in last 8 months until this week). Any pointers as to how to find
> the root cause would be greatly appreciated.
>
> Thanks in advance.
> Praveen
