We are running C* 2.0.15, recently 2 of our 10 nodes had to be forcefully removed. Cluster is behaving fine since then as we are not seeing any issues with production except that nodes every now and then throw out the following error:
Jun 19 17:18:35 cass-prod5.localdomain cassandra: 2015-06-19 17:18:35,091 INFO GossipStage:37 Gossiper.markDead - InetAddress /10.198.144.205 is now DOWN Jun 19 17:18:35 cass-prod5.localdomain cassandra: 2015-06-19 17:18:35,093 INFO GossipStage:37 StorageService.excise - Removing tokens [5854867837086423915] for /10.198.144.205 Jun 19 17:18:56 cass-prod5.localdomain cassandra: 2015-06-19 17:18:56,307 INFO GossipStage:37 Gossiper.markDead - InetAddress /10.238.2.149 is now DOWN Jun 19 17:18:56 cass-prod5.localdomain cassandra: 2015-06-19 17:18:56,308 INFO GossipStage:37 StorageService.excise - Removing tokens [7296642456956339369] for /10.238.2.149 Jun 19 17:19:41 cass-prod5.localdomain cassandra: 2015-06-19 17:19:41,108 INFO GossipStage:37 Gossiper.markDead - InetAddress /10.198.144.205 is now DOWN Jun 19 17:19:41 cass-prod5.localdomain cassandra: 2015-06-19 17:19:41,109 INFO GossipStage:37 StorageService.excise - Removing tokens [5854867837086423915] for /10.198.144.205 Jun 19 17:20:00 cass-prod5.localdomain cassandra: 2015-06-19 17:20:00,322 INFO GossipStage:37 Gossiper.markDead - InetAddress /10.238.2.149 is now DOWN Jun 19 17:20:00 cass-prod5.localdomain cassandra: 2015-06-19 17:20:00,324 INFO GossipStage:37 StorageService.excise - Removing tokens [7296642456956339369] for /10.238.2.149 Jun 19 17:20:00 cass-prod5.localdomain cassandra: 2015-06-19 17:20:00,329 ERROR GossipStage:37 CassandraDaemon.uncaughtException - Exception in thread Thread[GossipStage:37,5,main] Jun 19 17:20:00 cass-prod5.localdomain java.lang.NullPointerException Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.service.StorageService.getRpcaddress(StorageService.java:1067) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.transport.Server$EventNotifier.getRpcAddress(Server.java:345) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.transport.Server$EventNotifier.onLeaveCluster(Server.java:366) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.service.StorageService.excise(StorageService.java:1790) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.service.StorageService.excise(StorageService.java:1798) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1701) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1361) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1995) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1003) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1102) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49) Jun 19 17:20:00 cass-prod5.localdomain at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) Jun 19 17:20:00 cass-prod5.localdomain at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) Jun 19 17:20:00 cass-prod5.localdomain at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) Jun 19 17:20:00 cass-prod5.localdomain at java.lang.Thread.run(Thread.java:745) Each node says the same thing for the same 2 IPs. These IPs belong to the nodes we had to force remove. I have checked the systems.peers and no node is showing these 2 as their peers. Nodetool status is also showing the right boxes in ring. Not sure where these IPs are coming from in the cluster. Can someone throw any light on this please. Thanks
