I have an existing cluster of four Cassandra nodes. The machines have both an internal and an external IP, and originally I set them up to use the external network. A little while later I moved them to the internal network by bringing all machines down, changing the config, and bringing them up again. In the logs I found they all said "Changing ownership of token XXX", and nodetool ring reported that the cluster consisted of those four machines on their internal ips. After that, as part of a cleanup process, I moved the tokens on all machines to make sure the cluster was balanced, and it also worked perfectly.
However, now I have to temporarily move the cluster back to the external network for a little while. I tried doing the same thing as last time, bringing all nodes down, changing the config (rpc address, gossip address, list of seeds) and bringing them up again, but this resulted in a very confused cluster. When I ran nodetool ring, it reported eight nodes, the four internal ips were marked as down, and the four external were marked as up, but with the token they had when they previously used that ip. Checking the logs, there was no token ownership change, all nodes picked the saved token they had when they last used the external ip, and not the token they should have, the one I moved each server to when on the internal ip. I immediately moved all servers back to the internal IP, and then nodetool reported the same as before, a cluster of four machines, all up, and all on the token they're supposed to have. No mention of the external ips or the old tokens they had there. How do I reset this data? Where is it stored? Why does it store all of this when nodetool doesn't report it? Why does a node store several saved tokens? How do I change their ip without losing any data and without having to do removetoken or similar? One thought I have is to bring down one node, delete the system keyspace, and bring it back up, at which point it would only use what's in the config, but fetch the schema from the other nodes. Or would it also fetch the old information of what token it had when it was on the external ip? Or would something else go wrong? /Henrik
