> I'm not able to replace a dead node using the ordinary procedure 
> (boostrap+join), and would like to rebuild the replacement node from another 
> DC.
Normally when you want to add a new DC to the cluster the command to use is 
nodetool rebuild $DC_NAME .(with auto_bootstrap: false) That will get the node 
to stream data from the $DC_NAME

> The problem is that if I start a node with auto_bootstrap=false to perform 
> the rebuild, it automatically starts serving empty reads (CL=LOCAL_ONE).

When adding a new DC the nodes wont be processing reads, that is not the case 
for you.

You should disable the client API’s to prevent the clients from calling the new 
nodes, use -Dcassandra.start_rpc=false and 
-Dcassandra.start_native_transport=false in cassandra-env.sh or appropriate 
settings in cassandra.yaml

Disabling reads from other nodes will be harder. IIRC during bootstrap a 
different timeout (based on ring_delay) is used to detect if the bootstrapping 
node is down. However if the node is running and you use nodetool rebuild i’m 
pretty sure the normal gossip failure detectors will kick in. Which means you 
cannot disable gossip to prevent reads. Also we would want the node to be up 
for writes. 

But what you can do is artificially set the severity of the node high so the 
dynamic snitch will route around it. See 
https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/DynamicEndpointSnitchMBean.java#L37
 

* Set the value to something high on the node you will be rebuilding, the 
number or cores on the system should do.  (jmxterm is handy for this 
http://wiki.cyclopsgroup.org/jmxterm) 
* Check nodetool gossipinfo on the other nodes to see the SEVERITY app state 
has propagated. 
* Watch completed ReadStage tasks on the node you want to rebuild. If you have 
read repair enabled it will still get some traffic. 
* Do rebuild 
* Reset severity to 0

Hope that helps. 
Aaron

-----------------
Aaron Morton
New Zealand
@aaronmorton

Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

On 13/05/2014, at 5:18 am, Paulo Ricardo Motta Gomes 
<paulo.mo...@chaordicsystems.com> wrote:

> Hello,
> 
> I'm not able to replace a dead node using the ordinary procedure 
> (boostrap+join), and would like to rebuild the replacement node from another 
> DC. The problem is that if I start a node with auto_bootstrap=false to 
> perform the rebuild, it automatically starts serving empty reads 
> (CL=LOCAL_ONE).
> 
> Is there a way to disable reads from a node while performing rebuild from 
> another datacenter? I tried starting the node in write survery mode, but the 
> nodetool rebuild command does not work in this mode.
> 
> Thanks,
> 
> -- 
> Paulo Motta
> 
> Chaordic | Platform
> www.chaordic.com.br
> +55 48 3232.3200

Reply via email to