Hi Petr, >From the question, it looks like you are trying to find a shortcut to avoid longer restore times without having data loss. The TLDR answer to your question is that you cannot workaround by doing node level restore since the corrupted data will eventually get propagated. But there are some other ways you can make the restore quicker and ensure that you don't have any data loss.
1) Fine-grained Recovery (to save you time) If you know the column family(ies) or at least the keyspace(s) where the data corruption happened, you can save time by doing a fine-grained restore of only those column(ies) or keyspace(s). 2) Point-in-time Recovery (to prevent data loss) Assuming, you had a last correct snapshot, you can restore that version. However, all the newer data that came in after the snapshot and before the data corruption was introduced will still be lost. In order to avoid this, you will need to devise a point-in-time recovery strategy. 3) Repair-free Recovery (to save you time) By going back to last correct snapshot, you might undo the corruption, but you will also undo all the good work done by Cassandra via repairs since the snapshot was taken and before the corruption was introduced. For this, you will have to devise a repair-free recovery strategy. Hope this helps. Thank you ! Regards, Rajath ------------------------ Rajath Subramanyam On Fri, Dec 2, 2016 at 10:19 AM, Anuj Wadehra <anujw_2...@yahoo.co.in> wrote: > Hi Petr, > > If data corruption means accidental data deletions via Cassandra commands, > you have to restore entire cluster with latest snapshots. This may lead to > data loss as there may be valid updates after the snapshot was taken but > before the data deletion. Restoring single node with snapshot wont help as > Cassandra replicated the accidental deletes to all nodes. > > If data corruption means accidental deletion of some sstable files from > file system of a node, repair would fix it. > > If data corruption means unreadable data due to hardware issues etc, you > will have two options after replacing the disk: bootstrap or restore > snapshot on the single affected node. If you have huge data per node e.g. > 300Gb , you may want to restore from Snapshot followed by repair. Restoring > snapshot on single node is faster than streaming all data via bootstrap. > If the node is not recoverable and must be replaced, you should be able to > do auto-boostrap or restore from snapshot with auto-bootstrap set to false. > I havent replaced a dead node with snapshot but there should not be any > issues as token ranges dont change when you replace a node. > > > > Thanks > Anuj > > On Tue, 29 Nov, 2016 at 11:08 PM, Petr Malik > <pma...@tesora.com> wrote: > > > Hi. > > I have a question about Cassandra backup-restore strategies. > > As far as I understand Cassandra has been designed to survive hardware > failures by relying on data replication. > > > It seems like people still want backup/restore for case when somebody > accidentally deletes data or the data gets otherwise corrupted. > > In that case restoring all keyspace/table snapshots on all nodes should > bring it back. > > > I am asking because I often read directions on restoring a single node in > a cluster. I am just wondering under what circumstances could this be done > safely. > > > Please correct me if i am wrong but restoring just a single node does not > really roll back the data as the newer (corrupt) data will be served by > other replicas and eventually propagated to the restored node. Right? > > In fact by doing so one may end up reintroducing deleted data back... > > > Also since Cassandra distributes the data throughout the cluster it is not > clear on which mode any particular (corrupt) data resides and hence which > to restore. > > > I guess this is a long way of asking whether there is an advantage of > trying to restore just a single node in a Cassandra cluster as opposed to > say replacing the dead node and letting Cassandra handle the replication. > > > Thanks. > >
