Hello, Assuming you nodes are out for a while and you don't need the data after 60 days (or cannot get it anyway), the way to fix this is to force the node out. I would try, in this order:
- nodetool removenode HOSTID - nodetool removenode force These 2 might really not work at this stage, but if they do, this is a clean way to do so. Now, to really push the ghost nodes to the exit door, it often takes: - nodetool assassinate I think Cassandra 2.1 doesn't have it, you might have to use JMX, more details here: https://thelastpickle.com/blog/2018/09/18/assassinate.html): echo "run -b org.apache.cassandra.net:type=Gossiper > unsafeAssassinateEndpoint $IP_TO_ASSASSINATE" | java -jar > jmxterm-1.0.0-uber.jar -l $IP_OF_LIVE_NODE:7199 This should really remove the traces of the node, without any safety, no streaming, no checks, just get rid of it. So to use with a lot of care and understanding. In your situation I guess this is what will work. As a last attempt, you could try removing traces of the dead node(s) from all the live nodes 'system.peers' table. This table is local to each node, so the DELETE command is to be send to all the nodes (that have a trace of an old node). - cqlsh -e "DELETE $IP_TO_REMOVE FROM system.peers;" but I see the node IPs in UNREACHABLE state in "nodetool describecluster" > output. I believe they appear only for 72 hours, but in my case I see > those nodes in UNREACHABLE for ever (more than 60 days) To be more accurate, you should never see leaving node as unreachable I believe (not even for 72 hours). The 72 hours is the time Gossip should continue referencing the old nodes. Typically when you remove the ghost nodes, they should no longer appear in 'nodetool describe' cluster at all, I would say immediately, but still appear in 'nodetool gossipinfo' with a 'left' or 'remove' status. I hope that helps and that one of the above will do the trick (I'd bet on the assassinate :)). Also sorry it took us a while to answer you this relatively common question :); C*heers, ----------------------- Alain Rodriguez - al...@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com Le jeu. 13 juin 2019 à 00:55, Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> a écrit : > Hello, > > I have a Cassandra cluster running with 2.1.16 version of Cassandra, where > I have decommissioned few nodes from the cluster using "nodetool > decommission", but I see the node IPs in UNREACHABLE state in "nodetool > describecluster" output. I believe they appear only for 72 hours, but in > my case I see those nodes in UNREACHABLE for ever (more than 60 days). > Rolling restart of the nodes didn't remove them. any idea what could be > causing here? > > Note: I don't see them in the nodetool status output. >
