Dan, Thanks for the suggestions. Our startup script currently starts riak on each node, then goes back and joins each node to node 1, waiting until ringready is ok before advancing. Appropriate pauses added, and we had to reduce the gossip interval to 10 seconds to make it run quickly enough.
What we found was that 4 is apparently a minimum number of partitions per node. When we increased the partitions, we got better distribution up to 63 nodes. With 128 partitions, it stopped distributing them beyond the 42nd node added (and wasn't balancing on those nodes). Don't know if that is documented anywhere or not. I've seen recommendations for having more partitions, but we weren't aware there was a correctness issue. Bruce On Sep 30, 2010, at 4:16 PM, Dan Reverri wrote: > Hi Michael, > > There is no hard coded limit to the number of nodes that can belong to a > cluster. Depending on how quickly you are adding nodes to the cluster you may > be running into an issue with the ring not converging on a single state. > > Can you try using the "riak-admin ringready" command between each join of a > node? This command will return TRUE if the nodes all agree on a particular > ring state. You should run this command until it returns TRUE after each join. > > Also, Alexander is correct; you should deploy your nodes with an increased > value for the "ring_creation_size" parameter in app.config. The typical > recommendation is to have a minimum of 10 partitions per node and the value > should be a power of 2 (64, 128, 256, 512, 1024). For a 63 node cluster a > value of 1024 would work well. > > Thanks, > Dan > > Daniel Reverri > Developer Advocate > Basho Technologies, Inc. > d...@basho.com > > > On Thu, Sep 30, 2010 at 12:02 PM, Alexander Sicular <sicul...@gmail.com> > wrote: > The default vnode number is 64. If you are connecting 63 physical nodes > togeter in one riak cluster than you really want to increase the number of > vnodes. There is some recommended metric for physical nodes to vnodes. I > think it's like a minimum of 4 vnodes per physical node. But I could be > totally making that up. > > -Alexander. > > > @siculars on twitter > http://siculars.posterous.com > > Sent from my iPhone > > On Sep 30, 2010, at 14:26, Michael Colussi <mcolu...@gmail.com> wrote: > >> Hey guys, >> >> My team was trying to launch a 63 node Riak cluster yesterday, using version >> 0.13rc3 and the bitcask backend. >> >> All 63 nodes agreed that the ring consisted of nodes 1-42 only. >> >> Any ideas? Is there a hardcoded limit to the number of nodes allowed? >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
