Hi Tom, "one drawback: the node joins the cluster as soon as the bootstrapping begins." I am not sure I understand this correctly. It will get tokens, but not load data if you combine it with autobootstrap=false.
How I see it: You should be able to start all the new nodes in the new DC with autobootstrap=false and survey-mode=true. Then you should have a new DC with nodes that have tokens but no data. Then you can start rebuild on all new nodes. During this process, the new nodes should get writes, but not serve reads. Disclaimer: I have not tested the combination of the two! "It turns out that join_ring=false in this scenario does not solve this problem" I also don't see how joing_ring would help here. (Actually I have no clue where you would ever need that option) "A workaround could be to ensure that only LOCAL_* CL is used by all clients, but even then I'm seeing failed queries, because they're mysteriously routed to the new DC every now and then." Yes, it works fine if you don't do any mistakes. Keep in mind you also have to make sure your driver does not connect against the other DC. But I agree with you: its a workaround for this scenario. To me this does not feel correct. "Currently I'm trying to auto_bootstrap my new DC. The good thing is that it doesn't accept reads from other DCs." The joining-state actually works perfectly. The joining state is a state where node take writes, but not serve ready. It would be really cool if you could boot a node into the joining state. Actually, write_survey should basically be the same. kind regards, Christian PS: I would love to see the results, if you perform any tests on the write-survey. Please share it here on the mailing list :-) On Mon, Sep 7, 2015 at 11:10 PM, Tom van den Berge < tom.vandenbe...@gmail.com> wrote: > Hi Christian, > > No, I never tried survey mode. I didn't know it until now, but form the > info I was able to find it looks like it is meant for a different purpose. > Maybe it can be used to bootstrap a new DC, though. > > On the other hand, the auto_bootstrap=false + rebuild scenario seems to be > designed to do exactly what I need, except that it has one drawback: the > node joins the cluster as soon as the bootstrapping begins. > > It turns out that join_ring=false in this scenario does not solve this > problem, since nodetool rebuild does not do anything if C* is started with > this option. > > A workaround could be to ensure that only LOCAL_* CL is used by all > clients, but even then I'm seeing failed queries, because they're > mysteriously routed to the new DC every now and then. > > Currently I'm trying to auto_bootstrap my new DC. The good thing is that > it doesn't accept reads from other DCs. The bad thing is that a) I can't > choose where it streams its data from, and b) the two nodes I've been > trying to bootstrap crashed when they were almost finished... > > > > On Mon, Sep 7, 2015 at 10:22 PM, horschi <hors...@gmail.com> wrote: > >> Hi Tom, >> >> this sounds very much like my thread: "auto_bootstrap=false broken?" >> >> Did you try booting the new node with survey-mode? I wanted to try this, >> but I am waiting for 2.0.17 to come out (survey mode is broken in earlier >> versions). Imho survey mode is what you (and me too) want: start a node, >> accepting writes, but not serving reads. I have not tested it yet, but I >> think it should work. >> >> Also the manual join mentioned in CASSANDRA-9667 sounds very interesting. >> >> kind regards, >> Christian >> >> On Mon, Sep 7, 2015 at 10:11 PM, Tom van den Berge <t...@drillster.com> >> wrote: >> >>> Running nodetool rebuild on a node that was started with join_ring=false >>> does not work, unfortunately. The nodetool command returns immediately, >>> after a message appears in the log that the streaming of data has started. >>> After that, nothing happens. >>> >>> Tom >>> >>> >>> On Fri, Sep 12, 2014 at 5:47 PM, Robert Coli <rc...@eventbrite.com> >>> wrote: >>> >>>> On Fri, Sep 12, 2014 at 6:57 AM, Tom van den Berge <t...@drillster.com> >>>> wrote: >>>> >>>>> Wouldn't it be far more efficient if a node that is rebuilding itself >>>>> is responsible for not accepting reads until the rebuild is complete? E.g. >>>>> by marking it as "Joining", similar to a node that is being bootstrapped? >>>>> >>>> >>>> Yes, and Cassandra 2.0.7 and above contain this long desired >>>> functionality. >>>> >>>> https://issues.apache.org/jira/browse/CASSANDRA-6961 >>>> >>>> I presume that one can also run a rebuild in this state, though I >>>> haven't tried. Driftx gives it an 80% chance... try it and see and let us >>>> know? :D >>>> >>>> =Rob >>>> >>>> >>> >>> >>> >> >
