That's a nice workaround, will be really helpful in emergency situations like this.
Thanks, On Mon, May 12, 2014 at 6:58 PM, Aaron Morton <aa...@thelastpickle.com>wrote: > I'm not able to replace a dead node using the ordinary procedure > (boostrap+join), and would like to rebuild the replacement node from > another DC. > > Normally when you want to add a new DC to the cluster the command to use > is nodetool rebuild $DC_NAME .(with auto_bootstrap: false) That will get > the node to stream data from the $DC_NAME > > The problem is that if I start a node with auto_bootstrap=false to perform > the rebuild, it automatically starts serving empty reads (CL=LOCAL_ONE). > > When adding a new DC the nodes wont be processing reads, that is not the > case for you. > > You should disable the client API’s to prevent the clients from calling > the new nodes, use -Dcassandra.start_rpc=false and > -Dcassandra.start_native_transport=false in cassandra-env.sh or appropriate > settings in cassandra.yaml > > Disabling reads from other nodes will be harder. IIRC during bootstrap a > different timeout (based on ring_delay) is used to detect if the > bootstrapping node is down. However if the node is running and you use > nodetool rebuild i’m pretty sure the normal gossip failure detectors will > kick in. Which means you cannot disable gossip to prevent reads. Also we > would want the node to be up for writes. > > But what you can do is artificially set the severity of the node high so > the dynamic snitch will route around it. See > https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/DynamicEndpointSnitchMBean.java#L37 > > > * Set the value to something high on the node you will be rebuilding, the > number or cores on the system should do. (jmxterm is handy for this > http://wiki.cyclopsgroup.org/jmxterm) > * Check nodetool gossipinfo on the other nodes to see the SEVERITY app > state has propagated. > * Watch completed ReadStage tasks on the node you want to rebuild. If you > have read repair enabled it will still get some traffic. > * Do rebuild > * Reset severity to 0 > > Hope that helps. > Aaron > > ----------------- > Aaron Morton > New Zealand > @aaronmorton > > Co-Founder & Principal Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > > On 13/05/2014, at 5:18 am, Paulo Ricardo Motta Gomes < > paulo.mo...@chaordicsystems.com> wrote: > > Hello, > > I'm not able to replace a dead node using the ordinary procedure > (boostrap+join), and would like to rebuild the replacement node from > another DC. The problem is that if I start a node with auto_bootstrap=false > to perform the rebuild, it automatically starts serving empty reads > (CL=LOCAL_ONE). > > Is there a way to disable reads from a node while performing rebuild from > another datacenter? I tried starting the node in write survery mode, but > the nodetool rebuild command does not work in this mode. > > Thanks, > > -- > *Paulo Motta* > > Chaordic | *Platform* > *www.chaordic.com.br <http://www.chaordic.com.br/>* > +55 48 3232.3200 > > > -- *Paulo Motta* Chaordic | *Platform* *www.chaordic.com.br <http://www.chaordic.com.br/>* +55 48 3232.3200
