Is there any chance the replaced node recently resumed bootstrap, joined with join_ring=false or write_survey=true? If so, perhaps this could be related to CASSANDRA-12935.
Otherwise gossip tokens being empty is definitely unexpected behavior and you should probably file another ticket with more details/context (such as gossip debug logs of replacement and other nodes, and if the replacement node had the same or different ip as the original node since they are slightly different code paths after #8523). 2016-11-18 19:07 GMT-02:00 Dikang Gu <dikan...@gmail.com>: > Paulo, the tokens field for 2401:db00:2130:4091:face:0:13:0 shows > "TOKENS: not present", on all live nodes. It means tokens are missing, > right? What would cause this? > > Thanks. > Dikang. > > On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta <pauloricard...@gmail.com> > wrote: > >> What does nodetool gossipinfo shows for endpoint /2401:db00:2130:4091: >> face:0:13:0 ? Does it contain the TOKENS attribute? If it's missing, is >> it only missing on this node or other nodes as well? >> >> 2016-11-18 17:02 GMT-02:00 Dikang Gu <dikan...@gmail.com>: >> >>> Hi, I encountered couple times that I could not replace a down node due >>> to error: >>> >>> 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException) >>> encountered during startup: Could not find tokens for >>> /2401:db00:2130:4091:face:0:13:0 to replace >>> 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered >>> during startup >>> 2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find >>> tokens for /2401:db00:2130:4091:face:0:13:0 to replace >>> 2016-11-17_19:33:58.70491 at org.apache.cassandra.service.S >>> torageService.prepareReplacementInfo(StorageService.java:525) >>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >>> 160315.c29948b] >>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >>> torageService.prepareToJoin(StorageService.java:760) >>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >>> 160315.c29948b] >>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >>> torageService.initServer(StorageService.java:693) >>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >>> 160315.c29948b] >>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S >>> torageService.initServer(StorageService.java:585) >>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20 >>> 160315.c29948b] >>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.C >>> assandraDaemon.setup(CassandraDaemon.java:300) >>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >>> 60315.c29948b] >>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C >>> assandraDaemon.activate(CassandraDaemon.java:516) >>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >>> 60315.c29948b] >>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C >>> assandraDaemon.main(CassandraDaemon.java:625) >>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201 >>> 60315.c29948b] >>> 2016-11-17_19:33:58.70649 INFO 19:33:58 [StorageServiceShutdownHook]: >>> Announcing shutdown >>> 2016-11-17_19:34:00.70967 INFO 19:34:00 [StorageServiceShutdownHook]: >>> Waiting for messaging service to quiesce >>> 2016-11-17_19:34:00.71066 INFO 19:34:00 >>> [ACCEPT-/2401:db00:2130:4091:face:0:13:0]: >>> MessagingService has terminated the accept() thread >>> >>> Did not find a relevant ticket for this, is anyone aware of this? >>> >>> Thanks! >>> >>> -- >>> Dikang >>> >>> >> > > > -- > Dikang > >
