Hello Alain, Thanks for your suggestions.
Surprisingly, the node which is in unreachable state, is not present in any of the system tables. I am wondering, where the information is coming from. I checked system.peers for the IP in UNREACHABLE state and it's not present. I tried restart of Cassandra service as well. On Thu, Jun 20, 2019 at 5:59 AM Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Hello, > > Assuming you nodes are out for a while and you don't need the data after > 60 days (or cannot get it anyway), the way to fix this is to force the node > out. I would try, in this order: > > - nodetool removenode HOSTID > - nodetool removenode force > > These 2 might really not work at this stage, but if they do, this is a > clean way to do so. > Now, to really push the ghost nodes to the exit door, it often takes: > > - nodetool assassinate > > I think Cassandra 2.1 doesn't have it, you might have to use JMX, more > details here: https://thelastpickle.com/blog/2018/09/18/assassinate.html): > > echo "run -b org.apache.cassandra.net:type=Gossiper >> unsafeAssassinateEndpoint $IP_TO_ASSASSINATE" | java -jar >> jmxterm-1.0.0-uber.jar -l $IP_OF_LIVE_NODE:7199 > > > This should really remove the traces of the node, without any safety, no > streaming, no checks, just get rid of it. So to use with a lot of care and > understanding. In your situation I guess this is what will work. > > As a last attempt, you could try removing traces of the dead node(s) from > all the live nodes 'system.peers' table. This table is local to each node, > so the DELETE command is to be send to all the nodes (that have a trace of > an old node). > > - cqlsh -e "DELETE $IP_TO_REMOVE FROM system.peers;" > > but I see the node IPs in UNREACHABLE state in "nodetool describecluster" >> output. I believe they appear only for 72 hours, but in my case I see >> those nodes in UNREACHABLE for ever (more than 60 days) > > > To be more accurate, you should never see leaving node as unreachable I > believe (not even for 72 hours). The 72 hours is the time Gossip should > continue referencing the old nodes. Typically when you remove the ghost > nodes, they should no longer appear in 'nodetool describe' cluster at all, > I would say immediately, but still appear in 'nodetool gossipinfo' with a > 'left' or 'remove' status. > > I hope that helps and that one of the above will do the trick (I'd bet on > the assassinate :)). Also sorry it took us a while to answer you this > relatively common question :); > > C*heers, > ----------------------- > Alain Rodriguez - al...@thelastpickle.com > France / Spain > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > Le jeu. 13 juin 2019 à 00:55, Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> a écrit : > >> Hello, >> >> I have a Cassandra cluster running with 2.1.16 version of Cassandra, >> where I have decommissioned few nodes from the cluster using "nodetool >> decommission", but I see the node IPs in UNREACHABLE state in "nodetool >> describecluster" output. I believe they appear only for 72 hours, but in >> my case I see those nodes in UNREACHABLE for ever (more than 60 days). >> Rolling restart of the nodes didn't remove them. any idea what could be >> causing here? >> >> Note: I don't see them in the nodetool status output. >> >
