Hello, We have Cassandra cluster with 5 nodes hosted in the Amazon EC2, and I had to restart two of them, so their IPs changed. We use NetworkTopologyStrategy, so I simply updated IPs in the cassandra-topology.properties file.
However, as I understood, old IPs remained somewhere in the system keyspace, and now I observe several different exception stacktraces in the log files, including: java.lang.RuntimeException: Host ID collision between active endpoint /<new IP> and /<old IP> (id=ab66dd02-96b2-4504-8403-7d066f911698) at org.apache.cassandra.locator.TokenMetadata.updateHostId(TokenMetadata.java:229) at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1358) at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1228) at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1960) at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:837) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:915) at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:50) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) and java.lang.AssertionError: Missing host ID for <old IP> at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:583) at org.apache.cassandra.service.StorageProxy$5.runMayThrow(StorageProxy.java:552) at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1658) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) nodetool status being executed on 3 old nodes, shows old ghost node: Datacenter: DC1 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.14.128.109 2.8 GB 141 4.1% 32260392-12c2-4f1a-812e-87fd9a960d10 RAC2 UN 10.24.33.187 2.12 GB 258 42.7% ab66dd02-96b2-4504-8403-7d066f911698 RAC3 UN 10.20.149.165 2.99 GB 251 4.5% a0792f59-20b1-4017-a7f6-88e0c0d7f86f RAC1 DN 10.11.73.104 1.07 GB 2 1.0% null RAC1 Datacenter: DC2 =============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 10.34.78.23 2.21 GB 117 0.9% 2acd3766-404d-4cdc-b3e3-7b3b95856f0e RAC1 UN 10.20.23.171 2.22 GB 255 46.8% 67421e3a-1dfc-48a0-88b3-c6dbd64dc9d8 RAC1 Is it possible to fix those IP collisions ? Thanks.
