The article you mentioned here clearly says “For new users to Cassandra, the safest way to add multiple nodes into a cluster is to add them one at a time. Stay tuned as I will be following up with another post on bootstrapping.”
When extending cluster it is indeed recommended to go slow & serially. Optionally you can use cassandra.consistent.rangemovement=false but you can run in getting over streamed data. Since you’re using release way newer when fixed introduced , I assumed you won’t see same behavior as described for the version which fix addresses. After adding node , if you won’t get consistent data, you query consistency level should be able to pull consistent data , given you can tolerate bit latency until your repair is complete – if you go by recommendation i.e. to add one node at a time – you’ll avoid all these nuances . From: Fd Habash [mailto:fmhab...@gmail.com] Sent: Wednesday, May 01, 2019 3:12 PM To: user@cassandra.apache.org Subject: RE: Bootstrapping to Replace a Dead Node vs. Adding a New Node:Consistency Guarantees Probably, I needed to be clearer in my inquiry …. I’m investigating a situation where our diagnostic data is telling us that C* has lost some of the application data. I mean, getsstables for the data returns zero on all nodes in all racks. The last pickle article below & Jeff Jirsa had described a situation where bootstrapping a node to extend the cluster can loose data if this new node bootstraps from a stale SECONDARY replica (node that was offline > hinted had-off window). This was fixed in cassandra-2434. http://thelastpickle.com/blog/2017/05/23/auto-bootstrapping-part1.html<https://urldefense.proofpoint.com/v2/url?u=http-3A__thelastpickle.com_blog_2017_05_23_auto-2Dbootstrapping-2Dpart1.html&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=yAmuusU5W5Z8dvA1cs1QVdy7W9gg-vAXfeNGmQMPyy4&s=pbVHJ6KZhzTq8V70BIRPMwcBA9e3cfY6lWen1qDy8EU&e=> The article & the Jira above describe bootstrapping when extending a cluster. I understand replacing a dead node does not involve range movement, but will the above Jira fix prevent the bootstrap process when a replacing a dead node from using secondary replica? Thanks ---------------- Thank you From: Fred Habash<mailto:fmhab...@gmail.com> Sent: Wednesday, May 1, 2019 6:50 AM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: Re: Bootstrapping to Replace a Dead Node vs. Adding a New Node:Consistency Guarantees Thank you. Range movement is one reason this is enforced when adding a new node. But, what about forcing a consistent bootstrap i.e. bootstrapping from primary owner of the range and not a secondary replica. How’s consistent bootstrap enforced when replacing a dead node. ————- Thank you. On Apr 30, 2019, at 7:40 PM, Alok Dwivedi <alok.dwiv...@instaclustr.com<mailto:alok.dwiv...@instaclustr.com>> wrote: When a new node joins the ring, it needs to own new token ranges. This should be unique to the new node and we don’t want to end up in a situation where two nodes joining simultaneously can own same range (and ideally evenly distributed). Cassandra has this 2 minute wait rule for gossip state to propagate before a node is added. But this on its does not guarantees that token ranges can’t overlap. See this ticket for more details https://issues.apache.org/jira/browse/CASSANDRA-7069<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D7069&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=yAmuusU5W5Z8dvA1cs1QVdy7W9gg-vAXfeNGmQMPyy4&s=zDOzA0az-dhAjy3JStExiYNwTeizC1MJSRRAG-1NNyA&e=> To overcome this issue, the approach was to only allow one node joining at a time. When you replace a dead node the new token range selection does not applies as the replacing node just owns the token ranges of the dead node. I think that’s why the restriction of only replacing one node at a time does not applies in this case. Thanks Alok Dwivedi Senior Consultant https://www.instaclustr.com/platform/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.instaclustr.com_platform_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=yAmuusU5W5Z8dvA1cs1QVdy7W9gg-vAXfeNGmQMPyy4&s=bfUvw3cmdQCBT0el1ogPfMKVTFGOIzbJuKhaFtzKebw&e=> From: Fd Habash <fmhab...@gmail.com<mailto:fmhab...@gmail.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Wednesday, 1 May 2019 at 06:18 To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees Reviewing the documentation & based on my testing, using C* 2.2.8, I was not able to extend the cluster by adding multiple nodes simultaneously. I got an error message … Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true I understand this is to force a node to bootstrap from the former owner of the range when adding a node as part of extending the cluster. However, I was able to bootstrap multiple nodes to replace dead nodes. C* did not complain about it. Is consistent range movement & the guarantee it offers to bootstrap from primary range owner not applicable when bootstrapping to replace dead nodes? ---------------- Thank you
