Well that may explain why you do have an unbalanced distribution of data. It is not cassandra normal behaivor. If you lose your disk, and you don't have a backup of your data (at least system because it is not replicated), this is what I think should be the normal procedure to recover your node.
- stop cassandra on the node down - nodetool removenode id - update de conf of your cluster to remove the ip of the node removed (cassandra.yaml, cassandra-topology.properties). You might want to make a copy of your original conf - restart your cluster - check the node does not appear in nodetool status - restore your previous conf of your cluster. including the node that was down - on the node down, if its ip address is in the list of seeds, remove it in order to do the boobstrap - restart cassandra in every node one by one. It is what we do in our cluster. Saludos Jean Carlo "The best way to predict the future is to invent it" Alan Kay On Tue, Apr 26, 2016 at 2:25 PM, ssiv...@gmail.com <ssiv...@gmail.com> wrote: > No, I didn't. I just want to understand how C* handle such case and what > is a predictable behavior. > > > On 04/26/2016 02:51 PM, Jean Carlo wrote: > > Did you use a backup of the keyspace system? > > If not, you might do removenode of that node and re added to the cluster > to re generate new tokens. > > > Saludos > > Jean Carlo > > "The best way to predict the future is to invent it" Alan Kay > > On Tue, Apr 26, 2016 at 12:06 AM, <ssiv...@gmail.com>ssiv...@gmail.com < > ssiv...@gmail.com> wrote: > >> Hi All, >> >> I have cluster of 7 nodes completely balanced (each node owns ~500GB of >> data). >> And I have one keyspace and one table and three replicas. Than, I just >> failed one node's disk, replace it with a new one and started repairing. >> During that process I noticed that additional two nodes have started >> getting data, and at the end of the repairing three nodes have twice more >> data than at the beginning. >> I'm curious, is it a normal behavior for Cassandra? Why not only one >> node, but three, have gotten data during repairing? May be it's because of >> clocks skew? >> >> Thanks! >> >> -- >> best regards, >> Sergey >> >> > > -- > Thanks, > Serj > >
