> I'm not able to replace a dead node using the ordinary procedure > (boostrap+join), and would like to rebuild the replacement node from another > DC. Normally when you want to add a new DC to the cluster the command to use is nodetool rebuild $DC_NAME .(with auto_bootstrap: false) That will get the node to stream data from the $DC_NAME
> The problem is that if I start a node with auto_bootstrap=false to perform > the rebuild, it automatically starts serving empty reads (CL=LOCAL_ONE). When adding a new DC the nodes wont be processing reads, that is not the case for you. You should disable the client API’s to prevent the clients from calling the new nodes, use -Dcassandra.start_rpc=false and -Dcassandra.start_native_transport=false in cassandra-env.sh or appropriate settings in cassandra.yaml Disabling reads from other nodes will be harder. IIRC during bootstrap a different timeout (based on ring_delay) is used to detect if the bootstrapping node is down. However if the node is running and you use nodetool rebuild i’m pretty sure the normal gossip failure detectors will kick in. Which means you cannot disable gossip to prevent reads. Also we would want the node to be up for writes. But what you can do is artificially set the severity of the node high so the dynamic snitch will route around it. See https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/DynamicEndpointSnitchMBean.java#L37 * Set the value to something high on the node you will be rebuilding, the number or cores on the system should do. (jmxterm is handy for this http://wiki.cyclopsgroup.org/jmxterm) * Check nodetool gossipinfo on the other nodes to see the SEVERITY app state has propagated. * Watch completed ReadStage tasks on the node you want to rebuild. If you have read repair enabled it will still get some traffic. * Do rebuild * Reset severity to 0 Hope that helps. Aaron ----------------- Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 13/05/2014, at 5:18 am, Paulo Ricardo Motta Gomes <paulo.mo...@chaordicsystems.com> wrote: > Hello, > > I'm not able to replace a dead node using the ordinary procedure > (boostrap+join), and would like to rebuild the replacement node from another > DC. The problem is that if I start a node with auto_bootstrap=false to > perform the rebuild, it automatically starts serving empty reads > (CL=LOCAL_ONE). > > Is there a way to disable reads from a node while performing rebuild from > another datacenter? I tried starting the node in write survery mode, but the > nodetool rebuild command does not work in this mode. > > Thanks, > > -- > Paulo Motta > > Chaordic | Platform > www.chaordic.com.br > +55 48 3232.3200
