Ok, how about in this case I described? It runs out of memory with a single pair of nodes...
(Or did you mean there's a connection between each pair of vnodes?) On Wednesday, April 13, 2011 at 1:56 PM, Jon Meredith wrote: Hi Greg et al, > > As you say largest known is not largest possible. Internally within Basho, > the largest cluster we've experimented with so far had 50 nodes. > > Going beyond that it's speculation from me about pain points. > > 1) It is true that you need enough file descriptors to start up all > partitions when a node restarts - Riak checks if there is any handoff data > pending for each partition. We have work scheduled to address that in the > medium term. The plan is to only spin up partitions the node owns and any > that have been started as fallbacks that handoff has not completed for. Until > that work is done you will need a high ulimit with large ring sizes. > > 2) It is also true that Erlang runs a fully connected network, so there will > be connections between each node pair in the cluster. We haven't determined > the point at which it becomes a problem. > > So it looks like you'll be pushing the known limits. Basho will do our very > best to help overcome any obstacles as you encounter them. > > Jon Meredith > Basho Technologies. > > On Wed, Apr 13, 2011 at 1:41 PM, Greg Nelson <gro...@dropcam.com> wrote: > > The largest known riak cluster != the largest possible riak cluster. ;-) > > > > The inter node communication of the cluster depends on the data set and > > usage pattern, doesn't it? Or is there some constant overhead that tops out > > at a few hundred nodes? I should point out that we'll have big data, but > > not a huge number of keys. > > > > The number of vnodes in the cluster should be equal to the > > ring_creation_size under normal circumstances, shouldn't it? So when I have > > a one node cluster, that node is running ring_creation_size vnodes... File > > descriptors probably isn't a problem -- these machines won't be doing > > anything else, and the limits are set to 65536. > > > > Thinking about the internode communication you mentioned, that's probably > > where the resource hog is.. socket buffers, etc. > > > > Anyway, I'd also love to hear more from basho. :) > > On Wednesday, April 13, 2011 at 12:33 PM, sicul...@gmail.com wrote: > > > Ill just chime in and say that this is not practical for a few reasons. > > > The largest known riak cluster has like 50 or 60 nodes. Afaik, inter node > > > communication of erlang clusters top out at a few hundred nodes. I'm also > > > under the impression that each physical node has to have enough file > > > descriptors to accommodate every virtual node in the cluster. > > > > > > I'd love to hear more from basho. > > > > > > -alexander > > > > > > > > > Sent from my Verizon Wireless BlackBerry > > > > > > -----Original Message----- > > > From: Greg Nelson <gro...@dropcam.com> > > > Sender: riak-users-boun...@lists.basho.com > > > Date: Wed, 13 Apr 2011 12:13:34 > > > To: <riak-users@lists.basho.com> > > > Subject: Large ring_creation_size > > > > > > _______________________________________________ > > > riak-users mailing list > > > riak-users@lists.basho.com > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
