Hi, nodetool gossipinfo shows the decommissioned nodes as "LEFT"
I believe this is the expected behavior, we keep some a trace of leaving nodes for a few days, this shouldn't be an issue for you nodetool describecluster shows the decommissioned nodes as UNREACHABLE. > This is a weird behaviour I haven't see for a while. You might want to dig this some more. Restarting the entire cluster, everytime a node is decommissioned does not > seem right > Meanwhile, if you are sure the node is out and streams have ended, I guess it could be ok to use a JMX client (MX4J, JConsole...) and then use the JMX method Gossiper.unsafeAssassinateEndpoints(ip_address) to assassinate the gone node from any of the remaining nodes. How to --> http://tumblr.doki-pen.org/post/22654515359/assassinating-cassandra-nodes (3 years old post, I partially read it, but I think it might still be relevant) Has anybody experienced similar behaviour FTR, 3 years old similar issue I faced --> http://grokbase.com/t/cassandra/user/127knx7nn0/unreachable-node-not-in-nodetool-ring FWIW, people using C* = 3.x, this is exposed through nodetool --> https://docs.datastax.com/en/cassandra/3.x/cassandra/tools/toolsAssassinate.html Keep in mind that something called 'unsafe' and 'assassinate' at the same time is not something you want to use in a regular decommissioning process as it drop the node with no file transfer, you basically totally lose a node (unless node is out already which seems to be your case, it should be safe to use it in your case). I only used it to fix gossip status in the past or at some point when forcing a removenode was not working, followed by full repairs on remaining nodes. C*heers, ----------------- Alain Rodriguez France The Last Pickle http://www.thelastpickle.com 2016-02-16 20:08 GMT+01:00 sai krishnam raju potturi <pskraj...@gmail.com>: > hi; > we have a 12 node cluster across 2 datacenters. We are currently using > cassandra 2.1.12 version. > > SNITCH : GossipingPropertyFileSnitch > > When we decommissioned few nodes in a particular datacenter and observed > the following : > > nodetool status shows only the live nodes in the cluster. > > nodetool describecluster shows the decommissioned nodes as UNREACHABLE. > > nodetool gossipinfo shows the decommissioned nodes as "LEFT" > > > When the live nodes were restarted, "nodetool describecluster" shows only > the live nodes, which is expected. > > Purging the gossip info too did not help. > > INFO 17:27:07 InetAddress /X.X.X.X is now DOWN > INFO 17:27:07 Removing tokens [125897680671740685543105407593050165202, > 140213388002871593911508364312533329916, > 98576967436431350637134234839492449485] for /X.X.X.X > INFO 17:27:07 InetAddress /X.X.X.X is now DOWN > INFO 17:27:07 Removing tokens [11116977666116265389022494863106850615, > 111270759969411259938117902792984586225, > 138611464975439236357814418845450428175] for /X.X.X.X > > Has anybody experienced similar behaviour. Restarting the entire cluster, > everytime a node is decommissioned does not seem right. Thanks in advance > for the help. > > > thanks > Sai > > >
