Re: Unreachable node, not in nodetool ring

2012-08-08 Thread Alain RODRIGUEZ
Hi, I finally successfully removed the "ghost" node using unsafeAssassinateEndpoint() as described there : http://tumblr.doki-pen.org/post/22654515359/assinating-cassandra-nodes, I hope this can help more people. Nodetool gossipinfo gives me now the following info for the ghost node : /10.56.62.

Re: Unreachable node, not in nodetool ring

2012-07-27 Thread Olivier Mallassi
nope my last ideas would be (and I am not sure these are the best) - try removetoken with -f option. I do not believe it will change anything but... - try nodeltool ring on ALL nodes and check all nodes see the unreachable node. If not, you could maybe juste decommission the one(s) that see the

Re: Unreachable node, not in nodetool ring

2012-07-27 Thread Alain RODRIGUEZ
Hi again, Nobody has a clue about this issue ? I'm still facing this problem. Alain 2012/7/23 Alain RODRIGUEZ : > Does anyone knows how to totally remove a dead node that only appears > when doing a "describe cluster" from the cli ? > > I still got this issue in my production cluster. > > Alain

Re: Unreachable node, not in nodetool ring

2012-07-23 Thread Alain RODRIGUEZ
Does anyone knows how to totally remove a dead node that only appears when doing a "describe cluster" from the cli ? I still got this issue in my production cluster. Alain 2012/7/20 Alain RODRIGUEZ : > Hi Aaron, > > I have repaired and cleanup both nodes already and I did it after any > change o

Re: Unreachable node, not in nodetool ring

2012-07-20 Thread Alain RODRIGUEZ
Hi Aaron, I have repaired and cleanup both nodes already and I did it after any change on my ring (It tooks me a while btw :)). The node *.211 is actually out of the ring and out of my control 'cause I don't have the server anymore (EC2 instance terminated a few days ago). Alain 2012/7/20 aaron

Re: Unreachable node, not in nodetool ring

2012-07-20 Thread aaron morton
I would: * run repair on 10.58.83.109 * run cleanup on 10.59.21.241 (I assume this was the first node). It looks like 0.56.62.211 is out of the cluster. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 19/07/2012, at 9:37 PM, Alain RODRIG

Re: Unreachable node, not in nodetool ring

2012-07-19 Thread Alain RODRIGUEZ
Not sure if this may help : nodetool -h localhost gossipinfo /10.58.83.109 RELEASE_VERSION:1.1.2 RACK:1b LOAD:5.9384978406E10 SCHEMA:e7e0ec6c-616e-32e7-ae29-40eae2b82ca8 DC:eu-west STATUS:NORMAL,85070591730234615865843651857942052864 RPC_ADDRESS:0.0.0.0 /10.248.10.94 RELEASE_VERSIO

Re: Unreachable node, not in nodetool ring

2012-07-19 Thread Alain RODRIGUEZ
Hi, I wasn't able to see the token used currently by the 10.56.62.211 (ghost node). I already removed the token 6 days ago : -> "Removing token 170141183460469231731687303715884105727 for /10.56.62.211" "- check in cassandra log. It is possible you see a log line telling you 10.56.62.211 and 10.

Re: Unreachable node, not in nodetool ring

2012-07-19 Thread Olivier Mallassi
I got that a couple of time (due to DNS issues in our infra) what you could try - check in cassandra log. It is possible you see a log line telling you 10.56.62.211 and 10.59.21.241 o 10.58.83.109 share the same token - if 10.56.62.211 is up, try decommission (via nodetool) - if not, move 10.59.2

Unreachable node, not in nodetool ring

2012-07-19 Thread Alain RODRIGUEZ
Hi, I tried to add a node a few days ago and it failed. I finally made it work with an other node but now when I describe cluster on cli I got this : Cluster Information: Snitch: org.apache.cassandra.locator.Ec2Snitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema version