> It shouldn't remove a node from the ring should it? (appears it did)
It does. As explained here: http://wiki.apache.org/cassandra/Operations, loadbalance 'decomission' the node and then add it back as a bootstrapping node (roughly). So that the node disappear is expected and it is supposed to come back. But this is not a quick operation (and certainely not one you want to do every other day). You apparently restarted Cassandra while it was doing its stuff. Not sure the loss of data is to be expected though. > It shouldn't remove data from db, should it? (data size appears to grow, but > records are now missing) > > Loaded 38 million "rows" and the ring looked like this: > > m...@ec2:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host > 192.168.1.116 ring > Address Status Load Range > Ring > > 167730615856220406399741259265091647472 > 192.168.1.116 Up 4.81 GB > 54880762918591020775962843965839761529 |<--| > 192.168.1.119 Up 12.96 GB > 160455137948102479104219052453775170160 | | > 192.168.1.12 Up 8.98 GB > 167730615856220406399741259265091647472 |-- > > So I did this: > m...@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host > 192.168.1.12 loadbalance > > And this happened (even though Cassandra was still running): > > m...@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host > 192.168.1.12 ring > Address Status Load Range > Ring > > 160455137948102479104219052453775170160 > 192.168.1.116 Up 12.71 GB > 54880762918591020775962843965839761529 |<--| > 192.168.1.119 Up 13.47 GB > 160455137948102479104219052453775170160 |-->| > > After restarting Cassandra on .12 > > m...@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host > 192.168.1.12 ring > Address Status Load Range > Ring > > 160455137948102479104219052453775170160 > 192.168.1.116 Up 12.71 GB > 54880762918591020775962843965839761529 |<--| > 192.168.1.12 Up 8.98 GB > 107669873051407416105654071439122680093 | | > 192.168.1.119 Up 13.47 GB > 160455137948102479104219052453775170160 |-->| > > Now I have more data, but nearly 50% of my queries are failing (not found). > This data was checked before the load balance was done. >
