Re: Need help on dealing with Cassandra robustness and zombie data

2019-07-01 Thread Jeff Jirsa
What you’re describing is likely impossible to do in cassandra the way you’re thinking The only practical way to do it is extending gcgs and making the tombstone reads less expensive (ordering the clustering columns so you’re not scanning the tombstones, or breaking the partitions into buckets

Re: Need help on dealing with Cassandra robustness and zombie data

2019-07-01 Thread yuping wang
Thank you; very helpful. But we do have some difficulties #1 Cassandra process itself didn’t go down when marked as “DN”... (the node itself might just be temporary having some hiccup and not reachable )... so would not auto-start still help? #2 we can’t set longer gc grace because we are very se

Re: Need help on dealing with Cassandra robustness and zombie data

2019-07-01 Thread Rhys Campbell
#1 Set the cassandra service to not auto-start. #2 Longer gc_grace time would help #3 Rebootstrap? If the node doesn't come back within gc_grace,_seconds, remove the node, wipe it, and bootstrap it again. https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/dml/dml_about_deletes_c.html

Need help on dealing with Cassandra robustness and zombie data

2019-07-01 Thread yuping wang
Hi all, Sorry for the interruption. But I need help. Due to specific reasons of our use case, we have gc grace on the order of 10 minutes instead of default 10 days. Since we have a large amount of nodes in our Cassandra fleet, not surprisingly, we encounter occasionally node status go