As I asked this question I thought I would pipe in with my experience (comments inline).
On May 20, 2011, at 3:17 PM, Mark Phillips <m...@basho.com> wrote: > 4) Q --- Lets say I have several new nodes to add, is the recommended > procedure to add them one at a time and wait for all transfers to > finish, or can you actually add several? > > A --- The current recommended procedure is to add one node at a time > and wait for the partition transfers to finish before proceeding to > the next node addition. I found that adding them one at a time would have taken about 4 hours per node and as I was doubling the size I felt there would be less shuffling of data if I added all at once (as suggested by aphyr on IRC). This proved to be exactly correct as I was able to add 4 new nodes in about 4 hours instead of 16. > Specifically: > > * Use the "riak-admin join" command to kick off the cluster expansion > * Run "riak-admin transfers" periodically to keep an eye on the nodes > awaiting or passing off partitions (this may take a bit to complete); > an alternate (and less expensive) way to keep an eye on on this is to > just watch the logs. Running "riak-admin transfers" hardly ever works I would say it times out 95% of the time when attempting to add a new node. I don't know why this is and I hope it is fixed someday but I would recommend never running it. Unfortunately grepping logs is also tricky as you have to deal with lots of false positives if you done something like I did where you had a bunch of nodes crash then brought them up, only to realize you need to add capacity, so you add nodes. But now the logs on the first nodes have messages for transfers from the restart and the node addition. > * When "riak-admin ringready" prints "TRUE ..." to let you know that > all nodes agree on the ring state, you're good to go. This actually returned true before transfers were complete IIRC so I think this may not quite be right. > (It's worth nothing that making this process smoother and more fluid > is high on our list of priorities.) Good to know I look forward to this as I expect to be increasing my cluster up to close to 100 nodes by the end of this year. -Anthony _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
