On Tue, Nov 18, 2014 at 12:46 PM, Michael Shuler <mich...@pbandjelly.org> wrote:
> `nodetool cleanup` also looks interesting as an option. I don't understand why cleanup or scrub would help with a case where data is being un-tombstoned. " 1 November - column is deleted - gc_grace_period is 10 days 8 November - all 3 replicas have tombstone 13/14 November - column/tombstone is gone on 2 replicas, 3rd replica has the original value (!), with the original timestamp… " To me the smoking gun here is that 3rd replica has the original value. @OP : can you repro if you run a major compaction between the deletion and the tombstone collection? Basically, I am conjecturing that a compaction bug or one of the handful of "unmask previously deleted data" bugs are resulting in the unmasking of a non-tombstone row which is sitting in a SStable. OP could also support this conjecture by running sstablekeys on other SSTables on "3rd replica" and determining what masked values there are for the row prior to deletion. If the data is sitting in an old SStable, this is suggestive. One last question for OP would be whether the nodes were restarted during the time period this bug was observed. An assortment of the "unmask previously deleted data" bugs come from "dead" sstables in the data directory being marked "live" on a restart. =Rob http://twitter.com/rcolidba
