when we setup our new Cassandra Cluster recently, we looked at the problem of what to do when a node dies and needs to be replaced.
we could either let it be dead and have ops team spin up a new node and manually assign a token to it; we could also use Priam. but our company uses Puppet to manage nodes, instead of EC2 autoscaler, so Priam was more advanced than we need or could integrate. So we came up with a simple script that integrates with Puppet and "nventory" (Nventory.sourceforge.net), the idea is very simple, and similar to Priam. for example, we have 16 nodes, nventory makes sure that they are always given dns names like host1.mycompany.comhost2.mycompany....... host16.mycompany.com . if host10 dies and needs to be replaced, Puppet spins up a new box, and the new box acquires the hostname "host10.mycompany.com". When the new box boots up, it runs a script, which queries nventory to figure out its own hostname, and the hostname of its siblings in the nventory group (kind of similar to the asg idea), it uses its order in the 16 nodes to calculate its token. and does a "-Dcassandra.replace_token=......" when starting cassandra. the drawback is that you can do automatic launching of the new host. but that limitation is due to nventor/Puppet. Just want to share with the community here, cuz many people probably run Puppet+nventory, so this simple solution might work well Thanks Yang
