Good points, Dave. Also, it's worth mentioning that we've seen that many customers and open-source users think they will need many more nodes than they actually do. Many are able to start with 5 nodes and are happy for quite a while. The only way to tell what you actually need is to start with a baseline configuration and simulate some percentage above your current load. Once you've figured out what size that initial cluster is, start with (number of nodes) * 50 as the ring_creation_size (rounded to the nearest power of 2 of course). This gives you a growth factor of about 5 before you need to consider changing.
As well, there's some ops "common sense" that says the lifetime of any single architecture is 18 months or less. That doesn't necessarily mean that you'll be building a new cluster with a larger ring size in 18 months, but just that your needs will be different at that time and are hard to predict. Plan for now, worry about the 1000 node cluster when you actually need it. Sean Cribbs <s...@basho.com> Developer Advocate Basho Technologies, Inc. http://basho.com/ On Apr 14, 2011, at 9:09 AM, Dave Barnes wrote: > Sorry I feel compelled to chime in. > > Maybe you could assess your physical node limits and start with a small > configuration, then increase it and increase it until you hit a limit. > > Work small to large. > > Once you find the pain point, lets us know what resource ran out. > > You will learn a lot along the way on how your servers behave and we'll > discover a lot when you share the results. > > Thanks for digging in, > > Dave > > On Wed, Apr 13, 2011 at 5:11 PM, Greg Nelson <gro...@dropcam.com> wrote: > Ok, how about in this case I described? It runs out of memory with a single > pair of nodes... > > (Or did you mean there's a connection between each pair of vnodes?) > On Wednesday, April 13, 2011 at 1:56 PM, Jon Meredith wrote: > >> Hi Greg et al, >> >> As you say largest known is not largest possible. Internally within Basho, >> the largest cluster we've experimented with so far had 50 nodes. >> >> Going beyond that it's speculation from me about pain points. >> >> 1) It is true that you need enough file descriptors to start up all >> partitions when a node restarts - Riak checks if there is any handoff data >> pending for each partition. We have work scheduled to address that in the >> medium term. The plan is to only spin up partitions the node owns and any >> that have been started as fallbacks that handoff has not completed for. >> Until that work is done you will need a high ulimit with large ring sizes. >> >> 2) It is also true that Erlang runs a fully connected network, so there will >> be connections between each node pair in the cluster. We haven't determined >> the point at which it becomes a problem. >> >> So it looks like you'll be pushing the known limits. Basho will do our very >> best to help overcome any obstacles as you encounter them. >> >> Jon Meredith >> Basho Technologies. >> >> On Wed, Apr 13, 2011 at 1:41 PM, Greg Nelson <gro...@dropcam.com> wrote: >>> The largest known riak cluster != the largest possible riak cluster. ;-) >>> >>> The inter node communication of the cluster depends on the data set and >>> usage pattern, doesn't it? Or is there some constant overhead that tops >>> out at a few hundred nodes? I should point out that we'll have big data, >>> but not a huge number of keys. >>> >>> The number of vnodes in the cluster should be equal to the >>> ring_creation_size under normal circumstances, shouldn't it? So when I >>> have a one node cluster, that node is running ring_creation_size vnodes... >>> File descriptors probably isn't a problem -- these machines won't be doing >>> anything else, and the limits are set to 65536. >>> >>> Thinking about the internode communication you mentioned, that's probably >>> where the resource hog is.. socket buffers, etc. >>> >>> Anyway, I'd also love to hear more from basho. :) >>> On Wednesday, April 13, 2011 at 12:33 PM, sicul...@gmail.com wrote: >>> >>>> Ill just chime in and say that this is not practical for a few reasons. >>>> The largest known riak cluster has like 50 or 60 nodes. Afaik, inter node >>>> communication of erlang clusters top out at a few hundred nodes. I'm also >>>> under the impression that each physical node has to have enough file >>>> descriptors to accommodate every virtual node in the cluster. >>>> >>>> I'd love to hear more from basho. >>>> >>>> -alexander >>>> >>>> >>>> Sent from my Verizon Wireless BlackBerry >>>> >>>> -----Original Message----- >>>> From: Greg Nelson <gro...@dropcam.com> >>>> Sender: riak-users-boun...@lists.basho.com >>>> Date: Wed, 13 Apr 2011 12:13:34 >>>> To: <riak-users@lists.basho.com> >>>> Subject: Large ring_creation_size >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >> > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
