Because you were able to successfully run repair you can follow up with
a nodetool cleanup which will git rid of some of the extraneous data on
that (bigger) node. You're also assured after you run repair that
entropy beteen the nodes is minimal.
Assuming you're using the random ordered partitioner: To balance your
ring I would start by calculating the new token locations, then moving
each of your nodes backwards along their owned range to their new locations.
From the script on http://wiki.apache.org/cassandra/Operations your new
balanced tokens would be:
0
21267647932558653966460912964485513216
42535295865117307932921825928971026432
63802943797675961899382738893456539648
85070591730234615865843651857942052864
106338239662793269832304564822427566080
127605887595351923798765477786913079296
148873535527910577765226390751398592512
From this you can see that 10.46.108.{100, 101} is already in the
right place so you don't have to do anything with those nodes. Proceed
with moving 10.46.108.104 to its new token, the safest way to do this
would be to use nodetool move. Another way to do it could be to run a
remove-token followed by re-adding the node into the ring at its new
location. The risk here is that if you do not at least repair after
re-joining the ring (and before you move the next node in the ring) then
some of the data on that node would be ignored as it would now fall out
of the owned range, so it's good practice to immediately run repair on a
node that you do a removetoken / re-join on.
The rest of your balancing should be an iteration on the above steps
moving through the range.
On 6/9/11 6:21 AM, Jonathan Colby wrote:
I got myself into a situation where one node (10.47.108.100) has a lot more
data than the other nodes. In fact, the 1 TB disk on this node is almost
full. I added 3 new nodes and let cassandra automatically calculate new tokens
by taking the highest loaded nodes. Unfortunately there is still a big token
range this node is responsible for (5113... - 85070...). Yes, I know that
one option would be to rebalance the entire cluster with move but this is an
extremely time-consuming and error-prone process because of the amount of data
involved.
Our RF = 3 and we read/write quorum. The nodes have been repaired so I think
the data should be in good shape.
Question: Can I get myself out of this mess without installing new nodes? I was
thinking of either decommission or removetoken to have the cluster "rebalance
itself". The re-bootstrap this node to a new token.
Address Status State Load Owns Token
127605887595351923798765477786913079296
10.46.108.100 Up Normal 218.52 GB 25.00% 0
10.46.108.101 Up Normal 260.04 GB 12.50%
21267647932558653966460912964485513216
10.46.108.104 Up Normal 286.79 GB 17.56%
51138582157040063602728874106478613120
10.47.108.100 Up Normal 874.91 GB 19.94%
85070591730234615865843651857942052863
10.47.108.102 Up Normal 302.79 GB 4.16%
92156241323118845370666296304459139297
10.47.108.103 Up Normal 242.02 GB 4.16%
99241191538897700272878550821956884116
10.47.108.101 Up Normal 439.9 GB 8.34%
113427455640312821154458202477256070484
10.46.108.103 Up Normal 304 GB 8.33%
127605887595351923798765477786913079296
--
Ben Coverston
Director of Operations
DataStax -- The Apache Cassandra Company
http://www.datastax.com/
