That's a good point - a snapshot is certainly in order ASAP, if not already done.
One more thing I'd add about "data has to be consolidated from all the nodes" (from #3 below): - EITHER run the sstable2json ops on each node - OR if size permits, copy the relevant sstables (containing the desired keys, from the output of the nodetool getsstables) locally or onto a new single-node instance, start that instance and run the commands there If restoring the sstables from a snapshot, you'll need to do the latter anyway. Arvydas On Wed, Mar 8, 2017 at 1:55 PM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > DISCLAIMER: This is only my personal opinion. Evaluate the situation > carefully and if you find below suggestions useful, follow them at your own > risk. > > If I have understood the problem correctly, malicious deletes would > actually lead to deletion of data. I am not sure how everything is normal > after the deletes? > > If data is critical,you could: > > 1. Take a database snapshot immediately so that you dont lose information > if delete entrues in sstables are compacted together with original data. > > 2. Transfer snapshot to suitable place and Run some utility such as > sstabletojson to get the keys impacted by the deletes and original data for > keys. Data has to be consolidated from all the nodes. > > 3. Devise a strategy to restore deleted data. > > Thanks > Anuj > > > > On Tue, Mar 7, 2017 at 8:44 AM, Michael Fong > <michael.f...@ruckuswireless.com> wrote: > > Hi, all, > > > > > > We recently encountered an issue in production that some records were > mysteriously deleted with a timestamp 100+ years from now. Everything is > normal as of now, and how the deletion happened and accuracy of system > timestamp at that moment are unknown. We were wondering if there is a > general way to recover the mysteriously-deleted data when the timestamp > meta is screwed up. > > > > Thanks in advanced, > > > > Regards, > > > > Michael Fong > >
