Hi guys. Tonight one of our 3 nodes cluster has died. This is Cassandra 1.2. and RF = 3
After bringing up a new node and starting it with -Dreplace_address of the dead one different things are happening. On the new node nodetool status is showing the expected ring (him along with the other two working nodes), but his status is UN, when I would expect it to be UJ, because is joining, right? Also in this same node, the logs are not showing anything related to received streams, but streams are being received, the used disk space is growing and nodetool netstats shows progress. In this same node, this logs messages are continuously appearing: INFO [GossipStage:1] 2015-10-29 13:23:10,719 Gossiper.java (line 843) Node /<DEAD NODE IP> is now part of the cluster INFO [GossipStage:1] 2015-10-29 13:23:10,721 Gossiper.java (line 809) InetAddress /<DEAD NODE IP> is now UP WARN [GossipStage:1] 2015-10-29 13:23:10,723 StorageService.java (line 1469) Not updating token metadata for /<DEAD NODE IP> because I am replacing it INFO [GossipStage:1] 2015-10-29 13:23:10,723 StorageService.java (line 1567) Nodes /<DEAD NODE IP> and /<NEW NODE IP> have the same token 115915760983105627952720478187817787338. Ignoring /<DEAD NODE IP> INFO [GossipTasks:1] 2015-10-29 13:23:41,350 Gossiper.java (line 622) FatClient /<DEAD NODE IP> has been silent for 30000ms, removing from gossip Now switching to the old working nodes, nodetool status shows the old ring, with the failed node as DN, but the new node doesn't appear in any of them. Streams are flowing from one of them to the newcomer and both netstats and logs show it. Gossipinfo in the new node shows the two working nodes as normal and himself as hibernating and the other nodes show the same but also the dead node as normal too. Is that nodetool status/gossipinfo disagreement normal and the logs of the newcomer? Any experience with this? Regards Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
