Hi All, We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via 1.2.10). Probably after upgradesstable (but it's only a guess, because we noticed problem few weeks later), some rows became tombstoned. They just disappear from results of queries. After inverstigation I've noticed, that they are reachable via sstable2json. Example output for "non-existent" row:
{"key": "6e6e37716c6d665f6f61695f6463","metadata": {"deletionInfo": {"markedForDeleteAt":2201170739199,"localDeletionTime":0}},"columns": [["DATA","3c6f61695f64633a64(...)",1357677928108]]} ] If I understand correctly row is marked as deleted with timestamp in the far future, but it's still on the disk. Also localDeletionTime is set to 0, which may means, that it's kind of internal bug, not effect of client error. So my question is: is it true, that upgradesstable may do soemthing like that? How to find reasons for such strange cassandra behaviour? Is there any option of recovering such strange marked nodes? This problem touches about 500K rows of all 14M in our database, so the percentage is quite big. best regards Aleksander 2013-12-12 Robert Coli <rc...@eventbrite.com>: > On Wed, Dec 11, 2013 at 6:27 AM, Mathijs Vogelzang <math...@apptornado.com> > wrote: >> >> When I use sstable2json on the sstable on the destination cluster, it has >> "metadata": {"deletionInfo": >> {"markedForDeleteAt":1796952039620607,"localDeletionTime":0}}, whereas >> it doesn't have that in the source sstable. >> (Yes, this is a timestamp far into the future. All our hosts are >> properly synced through ntp). > > > This seems like a bug in sstableloader, I would report it on JIRA. > >> >> Naturally, copying the data again doesn't work to fix it, as the >> tombstone is far in the future. Apart from not having this happen at >> all, how can it be fixed? > > > Briefly, you'll want to purge that tombstone and then reload the data with a > reasonable timestamp. > > Dealing with rows with data (and tombstones) in the far future is described > in detail here : > > http://thelastpickle.com/blog/2011/12/15/Anatomy-of-a-Cassandra-Partition.html > > =Rob >