This isn't expected at all and there's definitely something wrong. However, when I restart last node (others remain down), when it comes up > "nodetool status" shows all down nodes under "DC1" no matter how long I > wait. >
This sounds like the node can't gossip with other nodes. My hypothesis is that cassandra-topology.properties gets loaded on startup. The file is used for PropertyFileSnitch and it's been known to cause intermittent gossip issues with GPFS. That's where the incorrect "DC1" comes from because the node doesn't know which DC the other nodes belong to. We haven't been able to replicate the issue so I don't know how it gets triggered. I've explained it in detail in a previous post [1]. If my hypothesis proves true, you can workaround the problem by deleting cassandra-topology.properties. When you restart Cassandra, the node should be back to normal. Cheers! [1] https://community.datastax.com/questions/4621/
