any inputs here? On Sat, May 2, 2020 at 12:49 PM Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote:
> Hello Alain, > > Thanks for your suggestions. > > Surprisingly, the node which is in unreachable state, is not present in > any of the system tables. I am wondering, where the information is coming > from. > I checked system.peers for the IP in UNREACHABLE state and it's not > present. I tried restart of Cassandra service as well. > > On Thu, Jun 20, 2019 at 5:59 AM Alain RODRIGUEZ <arodr...@gmail.com> > wrote: > >> Hello, >> >> Assuming you nodes are out for a while and you don't need the data after >> 60 days (or cannot get it anyway), the way to fix this is to force the node >> out. I would try, in this order: >> >> - nodetool removenode HOSTID >> - nodetool removenode force >> >> These 2 might really not work at this stage, but if they do, this is a >> clean way to do so. >> Now, to really push the ghost nodes to the exit door, it often takes: >> >> - nodetool assassinate >> >> I think Cassandra 2.1 doesn't have it, you might have to use JMX, more >> details here: https://thelastpickle.com/blog/2018/09/18/assassinate.html >> ): >> >> echo "run -b org.apache.cassandra.net:type=Gossiper >>> unsafeAssassinateEndpoint $IP_TO_ASSASSINATE" | java -jar >>> jmxterm-1.0.0-uber.jar -l $IP_OF_LIVE_NODE:7199 >> >> >> This should really remove the traces of the node, without any safety, no >> streaming, no checks, just get rid of it. So to use with a lot of care and >> understanding. In your situation I guess this is what will work. >> >> As a last attempt, you could try removing traces of the dead node(s) from >> all the live nodes 'system.peers' table. This table is local to each node, >> so the DELETE command is to be send to all the nodes (that have a trace of >> an old node). >> >> - cqlsh -e "DELETE $IP_TO_REMOVE FROM system.peers;" >> >> but I see the node IPs in UNREACHABLE state in "nodetool describecluster" >>> output. I believe they appear only for 72 hours, but in my case I see >>> those nodes in UNREACHABLE for ever (more than 60 days) >> >> >> To be more accurate, you should never see leaving node as unreachable I >> believe (not even for 72 hours). The 72 hours is the time Gossip should >> continue referencing the old nodes. Typically when you remove the ghost >> nodes, they should no longer appear in 'nodetool describe' cluster at all, >> I would say immediately, but still appear in 'nodetool gossipinfo' with a >> 'left' or 'remove' status. >> >> I hope that helps and that one of the above will do the trick (I'd bet on >> the assassinate :)). Also sorry it took us a while to answer you this >> relatively common question :); >> >> C*heers, >> ----------------------- >> Alain Rodriguez - al...@thelastpickle.com >> France / Spain >> >> The Last Pickle - Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> Le jeu. 13 juin 2019 à 00:55, Jai Bheemsen Rao Dhanwada < >> jaibheem...@gmail.com> a écrit : >> >>> Hello, >>> >>> I have a Cassandra cluster running with 2.1.16 version of Cassandra, >>> where I have decommissioned few nodes from the cluster using "nodetool >>> decommission", but I see the node IPs in UNREACHABLE state in "nodetool >>> describecluster" output. I believe they appear only for 72 hours, but in >>> my case I see those nodes in UNREACHABLE for ever (more than 60 days). >>> Rolling restart of the nodes didn't remove them. any idea what could be >>> causing here? >>> >>> Note: I don't see them in the nodetool status output. >>> >>
