Yes, I haven't run sstableloader. The data loss apperared somwhere on the line: 1.1.7-> 1.2.10 -> upgradesstable -> 2.0.2 -> normal operations ->2.0.3 normal operations -> now Today I've noticed that oldest files with broken values appear during repair (we do repair once a week on each node). Maybe it's the repair operation, which caused data loss? I've no idea. Currently our cluster is runing 2.0.3 version. We can do some tests on data to give you all info to track the bug. But our most crucial question is: can we recover loss, or should we start to think how to re-gather them? best regards Aleksander ps. I like your link Rob, i'll pin it over my desk ;) In Oracle there were a rule: never deploy RDBMS before release 2 ;)
2014-02-03 Robert Coli <rc...@eventbrite.com>: > On Mon, Feb 3, 2014 at 12:51 AM, olek.stas...@gmail.com > <olek.stas...@gmail.com> wrote: >> >> We've faced very similar effect after upgrade from 1.1.7 to 2.0 (via >> 1.2.10). Probably after upgradesstable (but it's only a guess, >> because we noticed problem few weeks later), some rows became >> tombstoned. > > > To be clear, you didn't run SSTableloader at all? If so, this is the > hypothetical case where normal streaming operations (replacing a node? what > streaming did you do?) results in data loss... > > Also, CASSANDRA-6527 is a good reminder regarding the following : > > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ > > =Rob
