Re: Bootstrap failure on C* 1.2.13

2014-05-14 Thread Paulo Ricardo Motta Gomes
Hello, After about 3 months I was able to solve this issue, which happened again after another node died. The problem is the datastax 1.2 node replacement docs [1] said that "This procedure applies to clusters using vnodes. If not using vnodes, use the instructions in the Cassandra 1.1 documentat

Re: Bootstrap failure on C* 1.2.13

2014-02-15 Thread Alain RODRIGUEZ
Hi Rob, I don't understand how setting those "initial_token" might solve this issue. Even more since we cannot set them before bootstrapping... Plus, once those tokens set, we would have to modify them after any new bootstrap / decommission. Which would also imply to run a rolling restart for the

Re: Bootstrap failure on C* 1.2.13

2014-02-14 Thread Robert Coli
On Fri, Feb 14, 2014 at 10:08 AM, Paulo Ricardo Motta Gomes < paulo.mo...@chaordicsystems.com> wrote: > But in our case, our cluster was not using VNodes, so this workaround will > probably not work with VNodes, since you cannot specify the 256 tokens from > the old node. > Sure you can, in a com

Re: Bootstrap failure on C* 1.2.13

2014-02-14 Thread Paulo Ricardo Motta Gomes
Hello Alain, I solved this with a brute force solution, but didn't understand exactly what happened behind the scenes. What I did was: a) removed the failed node from the ring with the unsafeAssassinate JMX option. b) this caused requests to that node to be routed to the following node which didn

Re: Bootstrap failure on C* 1.2.13

2014-02-14 Thread Alain RODRIGUEZ
Hi Paulo, Did you find out how to fix this issue ? I am experimenting the exact same issue after trying to help you on this exact subject a few days ago :). Config : 32 C*1.2.11 nodes, Vnodes enabled, RF=3, 1 DC, On AWS EC2 m1.xlarge. We added a few nodes (4) and it seems that this occurs on one

Re: Bootstrap failure on C* 1.2.13

2014-02-07 Thread Robert Coli
On Fri, Feb 7, 2014 at 4:41 AM, Alain RODRIGUEZ wrote: > From changelog : > > 1.2.15 > * Move handling of migration event source to solve bootstrap race > (CASSANDRA-6648) > > Maybe should you give this new version a try, if you suspect your issue to be > related to CASSANDRA-6648. > > 6648 ap

Re: Bootstrap failure on C* 1.2.13

2014-02-07 Thread Alain RODRIGUEZ
>From changelog : 1.2.15 * Move handling of migration event source to solve bootstrap race (CASSANDRA-6648) Maybe should you give this new version a try, if you suspect your issue to be related to CASSANDRA-6648. Hope this will solve your issue. 2014-02-06 16:48 GMT+01:00 Paulo Ricardo Motta

Bootstrap failure on C* 1.2.13

2014-02-06 Thread Paulo Ricardo Motta Gomes
Hello, One the nodes of our cluster failed and I performed the "replace a dead node procedure" described in [1]. After about 5 or 6 hours of streaming during bootstrap, the node fails with the exception "Unable to fetch range for keyspace foobar from any hosts." [2]. I haven't found any thread or