Hello, After about 3 months I was able to solve this issue, which happened again after another node died.
The problem is the datastax 1.2 node replacement docs [1] said that "This procedure applies to clusters using vnodes. If not using vnodes, use the instructions in the Cassandra 1.1 documentation". However, the 1.1 docs did not mention the property "-Dcassandra.replace_address=address_of_dead_node", which was only introduced in 1.2. So, what happens without this flag is that the replacement node tries to stream data from the dead node, failing the bootstrap process. Adding this flag solves the problem. Big thanks to driftx from #cassandra who helped troubleshoot the issue. The docs were already updated to mention the property even for non-vnodes cluster. [1] http://www.datastax.com/documentation/cassandra/1.2/cassandra/operations/ops_replace_node_t.html Cheers, On Sat, Feb 15, 2014 at 3:31 PM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Hi Rob, > > I don't understand how setting those "initial_token" might solve this > issue. Even more since we cannot set them before bootstrapping... > > Plus, once those tokens set, we would have to modify them after any new > bootstrap / decommission. Which would also imply to run a rolling restart > for the new configuration (cassandra.yaml) to be taken into account. This > is quite a heavy process to perform a "NOOP"... > > What did I miss ? > > Thanks for getting involved and trying to help anyway :). > > Alain > > > 2014-02-15 1:13 GMT+01:00 Robert Coli <rc...@eventbrite.com>: > > On Fri, Feb 14, 2014 at 10:08 AM, Paulo Ricardo Motta Gomes < >> paulo.mo...@chaordicsystems.com> wrote: >> >>> But in our case, our cluster was not using VNodes, so this workaround >>> will probably not work with VNodes, since you cannot specify the 256 tokens >>> from the old node. >>> >> >> Sure you can, in a comma delimited list. I plan to write a short blog >> post about this, but... >> >> I recommend that anyone using Cassandra, vnodes or not, always explicitly >> populate their initial_token line in cassandra.yaml. There are a number of >> cases where you will lose if you do not do so, and AFAICT no cases where >> you lose by doing so. >> >> If one is using vnodes and wants to do this, the process goes like : >> >> 1) set num_tokens to the desired number of vnodes >> 2) start node/bootstrap >> 3) use a one liner like jeffj's : >> " >> nodetool info -T | grep ^Token | awk '{ print $3 }' | tr \\n , | sed -e >> 's/,$/\n/' >> " >> to get a comma delimited list of the vnode tokens >> 4) insert this comma delimited list in initial_token, and comment out >> num_tokens (though it is a NOOP) >> >> =Rob >> > > -- *Paulo Motta* Chaordic | *Platform* *www.chaordic.com.br <http://www.chaordic.com.br/>* +55 48 3232.3200
