Re: Node replacement failed in 2.2

2016-11-21 Thread Dikang Gu
Hmm, I don't think we use join_ring=false or write_survey=true for that node. I already remove_node to take the bad node out of ring, will try to have more debug logs next time. Thanks. On Sun, Nov 20, 2016 at 2:31 PM, Paulo Motta wrote: > Is there any chance the replaced node recently resumed

Re: Node replacement failed in 2.2

2016-11-20 Thread Paulo Motta
Is there any chance the replaced node recently resumed bootstrap, joined with join_ring=false or write_survey=true? If so, perhaps this could be related to CASSANDRA-12935. Otherwise gossip tokens being empty is definitely unexpected behavior and you should probably file another ticket with more d

Re: Node replacement failed in 2.2

2016-11-18 Thread Dikang Gu
Paulo, the tokens field for 2401:db00:2130:4091:face:0:13:0 shows "TOKENS: not present", on all live nodes. It means tokens are missing, right? What would cause this? Thanks. Dikang. On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta wrote: > What does nodetool gossipinfo shows for endpoint /2401:db

Re: Node replacement failed in 2.2

2016-11-18 Thread Paulo Motta
What does nodetool gossipinfo shows for endpoint /2401:db00:2130:4091:face:0:13:0 ? Does it contain the TOKENS attribute? If it's missing, is it only missing on this node or other nodes as well? 2016-11-18 17:02 GMT-02:00 Dikang Gu : > Hi, I encountered couple times that I could not replace a dow

Node replacement failed in 2.2

2016-11-18 Thread Dikang Gu
Hi, I encountered couple times that I could not replace a down node due to error: 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException) encountered during startup: Could not find tokens for /2401:db00:2130:4091:face:0:13:0 to replace 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exce