Solved. Finally found that there was one node with 172.29.8.8 in gossipinfo
and appearing incomplete in status:
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UL 172.29.8.8 13.28 GB 256 9.5% null
1
Restarting this node removed the decommissioned node from gossip.
Jeff
On 3 July 2015 at 13:09, Jeff Williams <je...@wherethebitsroam.com> wrote:
> I'm thinking I'll have to find something similar for 2.0. I just don't
> understand where the node is coming from!
>
> Jeff
>
> On 1 July 2015 at 10:21, Vitalii Skakun <vitalii.ska...@gmail.com> wrote:
>
>> Hi,
>>
>> just a guess, there was a possibility to purge gossip state on a node, at
>> least in version 1.2
>>
>> http://docs.datastax.com/en/cassandra/1.2/cassandra/architecture/architectureGossipPurge_t.html
>>
>> the trick was to add -Dcassandra.load_ring_state=false somehow to the
>> jvm parameters
>>
>> I'm not sure if it works for 2.0 (since it's not in the doc for 2.x)
>>
>> 2015-07-01 10:23 GMT+03:00 Jeff Williams <je...@wherethebitsroam.com>:
>>
>>> Thanks for the tip Aiman, but this node is not in the seed list anywhere.
>>>
>>> Jeff
>>>
>>> On 30 June 2015 at 18:16, Aiman Parvaiz <ai...@flipagram.com> wrote:
>>>
>>>> I was having exactly the same issue with the same version, check your
>>>> seed list and make sure it contains only the live nodes, I know that seeds
>>>> are only read when cassandra starts but updating the seed list to live
>>>> nodes and then doing a roiling restart fixed this issue for me.
>>>> I hope this helps you.
>>>>
>>>> Thanks
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jun 30, 2015, at 4:42 AM, Jeff Williams <je...@wherethebitsroam.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have a cluster which had 4 datacenters running 2.0.12. Last week one
>>>> of the datacenters was decommissioned using nodetool decommission on each
>>>> of the servers in turn. This seemed to work fine until one of the nodes
>>>> started appearing in the logs of all of the remaining servers with messages
>>>> like:
>>>>
>>>> INFO [GossipStage:3] 2015-06-30 11:22:39,189 Gossiper.java (line 924)
>>>> InetAddress /172.29.8.8 is now DOWN
>>>> INFO [GossipStage:3] 2015-06-30 11:22:39,190 StorageService.java (line
>>>> 1773) Removing tokens [...] for /172.29.8.8
>>>>
>>>> These come up in the log every minute or two. I believe it may have
>>>> re-appeared after a repair, but I'm not sure.
>>>>
>>>> The problem is that this node does not exist in "nodetool status",
>>>> "nodetool gossipinfo" or in the system.peers table. So how can tell the
>>>> cluster that this node is decommissioned?
>>>>
>>>> Regards,
>>>> Jeff
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> ----
>> with best regards,
>> Vitalii Skakun
>>
>
>
