The log said Bootstrapping @ 07:34 (since it was 08:35, I assumed it wasn't doing anything, also, CPU usage was < 10%)
Turns out, when I restarted the node, it claimed the time was 7:35 rather than 8:35. Why would log4j be off by one hour? We are on CDT here, and have been for more than a week. The date command returns the appropriate time (Wed Apr 7 09:24:50 CDT 2010), I see no evidence of a TZ variable and /etc/timezone shows "America/Chicago" If it was off by 6 hours instead of 1, I could understand this, but its only off by one hour. System.getProperties() reports the timezone as blank Also, if the data is pushed out to the other nodes before the bootstrapping, why has data been lost? Does this mean that decommissioning a node results in data loss? -----Original Message----- From: Sylvain Lebresne [mailto:sylv...@yakaz.com] Sent: Wednesday, April 07, 2010 9:07 AM To: user@cassandra.apache.org Subject: Re: What is loadbalance supposed to do? 0.6.0RC1 > It shouldn't remove a node from the ring should it? (appears it did) It does. As explained here: http://wiki.apache.org/cassandra/Operations, loadbalance 'decomission' the node and then add it back as a bootstrapping node (roughly). So that the node disappear is expected and it is supposed to come back. But this is not a quick operation (and certainely not one you want to do every other day). You apparently restarted Cassandra while it was doing its stuff. Not sure the loss of data is to be expected though. > It shouldn't remove data from db, should it? (data size appears to grow, but > records are now missing) > > Loaded 38 million "rows" and the ring looked like this: > > m...@ec2:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host > 192.168.1.116 ring > Address Status Load Range > Ring > > 167730615856220406399741259265091647472 > 192.168.1.116 Up 4.81 GB > 54880762918591020775962843965839761529 |<--| > 192.168.1.119 Up 12.96 GB > 160455137948102479104219052453775170160 | | > 192.168.1.12 Up 8.98 GB > 167730615856220406399741259265091647472 |-- > > So I did this: > m...@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host > 192.168.1.12 loadbalance > > And this happened (even though Cassandra was still running): > > m...@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host > 192.168.1.12 ring > Address Status Load Range > Ring > > 160455137948102479104219052453775170160 > 192.168.1.116 Up 12.71 GB > 54880762918591020775962843965839761529 |<--| > 192.168.1.119 Up 13.47 GB > 160455137948102479104219052453775170160 |-->| > > After restarting Cassandra on .12 > > m...@record:~/cassandra/apache-cassandra-0.6.0-rc1$ bin/nodetool --host > 192.168.1.12 ring > Address Status Load Range > Ring > > 160455137948102479104219052453775170160 > 192.168.1.116 Up 12.71 GB > 54880762918591020775962843965839761529 |<--| > 192.168.1.12 Up 8.98 GB > 107669873051407416105654071439122680093 | | > 192.168.1.119 Up 13.47 GB > 160455137948102479104219052453775170160 |-->| > > Now I have more data, but nearly 50% of my queries are failing (not found). > This data was checked before the load balance was done. >
