On Tue, Jan 10, 2012 at 11:17 AM, Elias Levy <fearsome.lucid...@gmail.com> wrote:
> Is this expected behavior? Granted, these are EC2 boxes and Leveldb depends > heavily on disk, but I can't imagine folks using Riak on production his this > type of performance hit resulting from rebalancing. tl;dr - yes, adding nodes can create a lot of load on the system. We've done a lot of work in this area recently. Rebalancing can be quite expensive with the default claim algorithm that ships in all releases to date. In the 1.0.3 release (dropping this week, I hope), we will be switching to a new default claim algorithm that will dramatically decrease the amount of rebalancing/shuffling that will be required. With this change in place, we've seen production clusters grow from 11 to 30+ nodes with minimal impacts. One of the other areas where we've seen systems get overloaded when adding nodes is that there is too much concurrent handoff. We've addressed this (also in 1.0.3) by tuning down the default setting for (outbound) handoff_concurrency (was 4, now 1). In addition, the 1.1 release (mid-late Feb) will ship with ability to limit inbound handoff_concurrency (you can try this now on HEAD). It's also worth noting that adding a 4th node to a 3 node cluster is going to be way more expensive/impactful than adding a 5th node. In a 3-node cluster, with N=3, every node is involved in every request -- add handoff on top of that and latency will get crazy. > I recall there was some mention that by using Riak EDS it was possible to > fill in a new node slowly, and then have it join the cluster when its nearly > in sync, thus limiting the stampede effect that regular rebalancing seems to > bring on. Is that the case? If so, is there any documentation on this? No, all the handoff/rebalancing stuff is part of our core product and, as such, is completely Open Source. handoff_concurrency + better claim algo is the way to manage this stampede. > Or are the time stats not rolling stats? Are they computed since the start > of the node, meaning that the 100 percentile will never go down, only up? Yes, they should be rolling stats on a 1 minute interval. We've observed normal behaviour in the production systems we've seen, but I'll ask someone to double check that they are working correctly. Thanks for reporting this. D. -- Dave Smith Director, Engineering Basho Technologies, Inc. diz...@basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
