Hi In the Cassandra version 3.10, after we decommission a node or datacenter, we observe the decommissioned nodes marked as DOWN in the cluster when you do a "nodetool describecluster". The nodes however do not show up in the "nodetool status" command. The decommissioned node also does not show up in the "system_peers" table on the nodes.
The workaround we follow is rolling restart of the cluster, which removes the decommissioned nodes from the "UNREACHABLE STATE", and shows the actual state of the cluster. The workaround is tedious for huge clusters. as anybody in the community observed similar issue? Below are the observed logs 2017-06-12 18:23:29,209 [RMI TCP Connection(8)-127.0.0.1] INFO StorageService.java:3938 - Announcing that I have left the ring for 30000ms 2017-06-12 18:23:59,210 [RMI TCP Connection(8)-127.0.0.1] INFO ThriftServer.java:139 - Stop listening to thrift clients 2017-06-12 18:23:59,215 [RMI TCP Connection(8)-127.0.0.1] INFO Server.java:176 - Stop listening for CQL clients 2017-06-12 18:23:59,216 [RMI TCP Connection(8)-127.0.0.1] WARN Gossiper.java:1514 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown 2017-06-12 18:23:59,216 [RMI TCP Connection(8)-127.0.0.1] INFO MessagingService.java:964 - Waiting for messaging service to quiesce 2017-06-12 18:23:59,217 [ACCEPT-/96.115.209.228] INFO MessagingService.java:1314 - MessagingService has terminated the accept() thread 2017-06-12 18:23:59,263 [RMI TCP Connection(8)-127.0.0.1] INFO StorageService.java:1435 - DECOMMISSIONED Regards, Avinash.
