Thank you; very helpful. But we do have some difficulties #1 Cassandra process itself didn’t go down when marked as “DN”... (the node itself might just be temporary having some hiccup and not reachable )... so would not auto-start still help? #2 we can’t set longer gc grace because we are very sensitive to latency ... and we have a lot data in and data out... so we can’t afford keep that large tombstone #3 the question what is the reliable way to detect change of node status? We tried to use a crontab job to poll nodestatus every 5 minutes... but we still end up missing some change of status especially if the node is bouncing up and down... also by the time we detect and try to replace node permanently, we might already exceeded that grace period.
Thanks again, Yuping On Jul 1, 2019, at 9:02 AM, Rhys Campbell <rhys.james.campb...@googlemail.com.invalid> wrote: #1 Set the cassandra service to not auto-start. #2 Longer gc_grace time would help #3 Rebootstrap? If the node doesn't come back within gc_grace,_seconds, remove the node, wipe it, and bootstrap it again. https://docs.datastax.com/en/archived/cassandra/2.0/cassandra/dml/dml_about_deletes_c.html yuping wang <yupingwyp1...@gmail.com> schrieb am Mo., 1. Juli 2019, 13:33: > Hi all, > > Sorry for the interruption. But I need help. > > > Due to specific reasons of our use case, we have gc grace on the order of > 10 minutes instead of default 10 days. Since we have a large amount of nodes > in our Cassandra fleet, not surprisingly, we encounter occasionally node > status going from up to down and up again. The problem is when the down node > rejoins the cluster after 15 minutes, it automatically adds already deleted > data back and causing zombie data. > our questions: > Is there a way to not allow a down node to rejoin the cluster? > or is there a way to configure rejoining node not adding stale data back > regardless of how long the node is down before rejoining > or is there a way to auto clean up the data when rejoining ? > We know adding those data back is a conservative approach to avoid data loss > but in our specific case, we are not worried about deleted data being > revived.... we don’t have such use case. We really need a non-defaul option > to never add back deleted data on rejoining nodes. > this functionality will ultimately be a deciding factor on whether we can > continue with Cassandra. > > Thanks again,
