Hmm, I don't think we use join_ring=false or write_survey=true for that node. I already remove_node to take the bad node out of ring, will try to have more debug logs next time.
Thanks. On Sun, Nov 20, 2016 at 2:31 PM, Paulo Motta <pauloricard...@gmail.com> wrote: > Is there any chance the replaced node recently resumed bootstrap, joined > with join_ring=false or write_survey=true? If so, perhaps this could be > related to CASSANDRA-12935. > > Otherwise gossip tokens being empty is definitely unexpected behavior and > you should probably file another ticket with more details/context (such as > gossip debug logs of replacement and other nodes, and if the replacement > node had the same or different ip as the original node since they are > slightly different code paths after #8523). > > 2016-11-18 19:07 GMT-02:00 Dikang Gu <dikan...@gmail.com>: > >> Paulo, the tokens field for 2401:db00:2130:4091:face:0:13:0 shows >> "TOKENS: not present", on all live nodes. It means tokens are missing, >> right? What would cause this? >> >> Thanks. >> Dikang. >> >> On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta <pauloricard...@gmail.com> >> wrote: >> >>> What does nodetool gossipinfo shows for endpoint /2401:db00:2130:4091: >>> face:0:13:0 ? Does it contain the TOKENS attribute? If it's missing, is >>> it only missing on this node or other nodes as well? >>> >>> 2016-11-18 17:02 GMT-02:00 Dikang Gu <dikan...@gmail.com>: >>> >>>> Hi, I encountered couple times that I could not replace a down node due >>>> to error: >>>> >>>> 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException) >>>> encountered during startup: Could not find tokens for >>>> /2401:db00:2130:4091:face:0:13:0 to replace >>>> 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered >>>> during startup >>>> 2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find >>>> tokens for /2401:db00:2130:4091:face:0:13:0 to replace >>>> 2016-11-17_19:33:58.70491 at org.apache.cassandra.service.S >>>> torageService.prepareReplacementInfo(StorageService.java:525) >>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >>>> 160315.c29948b] >>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >>>> torageService.prepareToJoin(StorageService.java:760) >>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >>>> 160315.c29948b] >>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >>>> torageService.initServer(StorageService.java:693) >>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >>>> 160315.c29948b] >>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >>>> torageService.initServer(StorageService.java:585) >>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >>>> 160315.c29948b] >>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.C >>>> assandraDaemon.setup(CassandraDaemon.java:300) >>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >>>> 60315.c29948b] >>>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C >>>> assandraDaemon.activate(CassandraDaemon.java:516) >>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >>>> 60315.c29948b] >>>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C >>>> assandraDaemon.main(CassandraDaemon.java:625) >>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >>>> 60315.c29948b] >>>> 2016-11-17_19:33:58.70649 INFO 19:33:58 [StorageServiceShutdownHook]: >>>> Announcing shutdown >>>> 2016-11-17_19:34:00.70967 INFO 19:34:00 [StorageServiceShutdownHook]: >>>> Waiting for messaging service to quiesce >>>> 2016-11-17_19:34:00.71066 INFO 19:34:00 >>>> [ACCEPT-/2401:db00:2130:4091:face:0:13:0]: >>>> MessagingService has terminated the accept() thread >>>> >>>> Did not find a relevant ticket for this, is anyone aware of this? >>>> >>>> Thanks! >>>> >>>> -- >>>> Dikang >>>> >>>> >>> >> >> >> -- >> Dikang >> >> > -- Dikang
