I, probably, should've been clearer in my inquiry ... I'm investigating a scenario where our diagnostic data is tell us that a small portion of application data has been lost. I mean, getsstables for the keys returns zero on all cluster nodes.
The last pickle article below (which includes a case scenario described by Jeff Jirsa) suggests possible data loss case when bootstrapping a new node to extend the cluster. The new node may bootstrap from a stale SECONDARY replica. A fix was made in Cassandra-2434. However, the article, the Jira, and Jeff's example all describe the scenario when extending a cluster. I understand replacing a dead node does not involve range movement. But, will the above fix force the bootstrap happening while replacing a dead node from streaming the data from a secondary (potentially) stale node? In other words, that fact that I was able to bootstrap multiple dead nodes, does it mean it is safe to do so? http://thelastpickle.com/blog/2017/05/23/auto-bootstrapping-part1.html Thanks On Tue, Apr 30, 2019 at 7:41 PM Alok Dwivedi <alok.dwiv...@instaclustr.com> wrote: > When a new node joins the ring, it needs to own new token ranges. This > should be unique to the new node and we don’t want to end up in a situation > where two nodes joining simultaneously can own same range (and ideally > evenly distributed). Cassandra has this 2 minute wait rule for gossip state > to propagate before a node is added. But this on its does not guarantees > that token ranges can’t overlap. See this ticket for more details > https://issues.apache.org/jira/browse/CASSANDRA-7069 To overcome this > issue, the approach was to only allow one node joining at a time. > > > > When you replace a dead node the new token range selection does not > applies as the replacing node just owns the token ranges of the dead node. > I think that’s why the restriction of only replacing one node at a time > does not applies in this case. > > > > > > Thanks > > Alok Dwivedi > > Senior Consultant > > https://www.instaclustr.com/platform/ > > > > > > > > > > > > *From: *Fd Habash <fmhab...@gmail.com> > *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Date: *Wednesday, 1 May 2019 at 06:18 > *To: *"user@cassandra.apache.org" <user@cassandra.apache.org> > *Subject: *Bootstrapping to Replace a Dead Node vs. Adding a New Node: > Consistency Guarantees > > > > Reviewing the documentation & based on my testing, using C* 2.2.8, I was > not able to extend the cluster by adding multiple nodes simultaneously. I > got an error message … > > > > Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while > cassandra.consistent.rangemovement is true > > > > I understand this is to force a node to bootstrap from the former owner of > the range when adding a node as part of extending the cluster. > > > > However, I was able to bootstrap multiple nodes to replace dead nodes. C* > did not complain about it. > > > > Is consistent range movement & the guarantee it offers to bootstrap from > primary range owner not applicable when bootstrapping to replace dead > nodes? > > > > ---------------- > Thank you > > > -- ---------------------------------------- Thank you
