This might be useful Nodetool command to disable reads<https://issues.apache.org/jira/browse/CASSANDRA-6760>
On Wed, May 14, 2014 at 8:31 AM, Paulo Ricardo Motta Gomes < paulo.mo...@chaordicsystems.com> wrote: > That's a nice workaround, will be really helpful in emergency situations > like this. > > Thanks, > > > On Mon, May 12, 2014 at 6:58 PM, Aaron Morton <aa...@thelastpickle.com>wrote: > >> I'm not able to replace a dead node using the ordinary procedure >> (boostrap+join), and would like to rebuild the replacement node from >> another DC. >> >> Normally when you want to add a new DC to the cluster the command to use >> is nodetool rebuild $DC_NAME .(with auto_bootstrap: false) That will get >> the node to stream data from the $DC_NAME >> >> The problem is that if I start a node with auto_bootstrap=false to >> perform the rebuild, it automatically starts serving empty reads >> (CL=LOCAL_ONE). >> >> When adding a new DC the nodes wont be processing reads, that is not >> the case for you. >> >> You should disable the client API’s to prevent the clients from calling >> the new nodes, use -Dcassandra.start_rpc=false and >> -Dcassandra.start_native_transport=false in cassandra-env.sh or appropriate >> settings in cassandra.yaml >> >> Disabling reads from other nodes will be harder. IIRC during bootstrap a >> different timeout (based on ring_delay) is used to detect if the >> bootstrapping node is down. However if the node is running and you use >> nodetool rebuild i’m pretty sure the normal gossip failure detectors will >> kick in. Which means you cannot disable gossip to prevent reads. Also we >> would want the node to be up for writes. >> >> But what you can do is artificially set the severity of the node high so >> the dynamic snitch will route around it. See >> https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/DynamicEndpointSnitchMBean.java#L37 >> >> >> * Set the value to something high on the node you will be rebuilding, the >> number or cores on the system should do. (jmxterm is handy for this >> http://wiki.cyclopsgroup.org/jmxterm) >> * Check nodetool gossipinfo on the other nodes to see the SEVERITY app >> state has propagated. >> * Watch completed ReadStage tasks on the node you want to rebuild. If you >> have read repair enabled it will still get some traffic. >> * Do rebuild >> * Reset severity to 0 >> >> Hope that helps. >> Aaron >> >> ----------------- >> Aaron Morton >> New Zealand >> @aaronmorton >> >> Co-Founder & Principal Consultant >> Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> On 13/05/2014, at 5:18 am, Paulo Ricardo Motta Gomes < >> paulo.mo...@chaordicsystems.com> wrote: >> >> Hello, >> >> I'm not able to replace a dead node using the ordinary procedure >> (boostrap+join), and would like to rebuild the replacement node from >> another DC. The problem is that if I start a node with auto_bootstrap=false >> to perform the rebuild, it automatically starts serving empty reads >> (CL=LOCAL_ONE). >> >> Is there a way to disable reads from a node while performing rebuild from >> another datacenter? I tried starting the node in write survery mode, but >> the nodetool rebuild command does not work in this mode. >> >> Thanks, >> >> -- >> *Paulo Motta* >> >> Chaordic | *Platform* >> *www.chaordic.com.br <http://www.chaordic.com.br/>* >> +55 48 3232.3200 >> >> >> > > > -- > *Paulo Motta* > > Chaordic | *Platform* > *www.chaordic.com.br <http://www.chaordic.com.br/>* > +55 48 3232.3200 >
