Chris,
I've spoken to our performance-tuning experts and unfortunately they say
there's not too much we can do with 1.4 to eek out more bandwidth for
handoff. However, the two settings you might try:
{riak_core, [
{handoff_batch_threshold, 1048576},
{handoff_concurrency, 2} ]}
handoff_batch_threshold is how big of a buffer it will collect before doing
a TCP send. You could bump that up to, say, 4MB (4 * 1024 * 1024), but we
have found it to be somewhat inconclusive in tests.
The other option (which can be set at runtime using riak-admin) is
handoff_concurrency, which is how many handoffs (incoming + outgoing) can
be running at a time. Increasing this will also increase disk IO
utilization, so tune it carefully.
We are looking at a lot of ways to increase performance in Riak 2.0, but as
many performance things go, the bottlenecks are not always what we first
think. I'm sure our perf experts will be able to discuss this more as 2.0
approaches.
Sorry I can't provide a more satisfactory answer.
On Mon, Oct 28, 2013 at 9:59 AM, Chris Read <chris.r...@gmail.com> wrote:
> Yes, we have have jumbo frames on.
>
> Kernel and systems are in a good state. I've got the settings from the
> Linux perf tuning guide on the riak docs. Testing with iperf(1) shows our
> nodes can saturate the 10G interfaces without breaking a sweat - as long as
> the packets are large enough...
>
> 1) http://iperf.sourceforge.net/
>
>
> On Sat, Oct 26, 2013 at 11:36 AM, Sean Cribbs <s...@basho.com> wrote:
>
>> Chris,
>>
>> Are you using jumbo frames on your network? (MTU 9000 or w/e) That's just
>> one aspect, but we've also found there are many other kernel/TCP-stack
>> tunings that need to be done to get Riak to push the limits of the network.
>>
>>
>> On Fri, Oct 25, 2013 at 9:10 PM, Chris Read <chris.r...@gmail.com> wrote:
>>
>>> Greetings all...
>>>
>>> Context: We're running Riak 1.4.2 on Ubuntu 12.04 using the Basho build
>>> .deb package. Out object size varies between 1K and 512K
>>>
>>> I've been having some trouble trying to figure out why handoffs between
>>> our nodes takes so long and never pushes the network card over 2.5 Gb/s.
>>> Testing communication between nodes using iperf shows that the
>>> configuration we have can sustain 10G.
>>>
>>> Further investigation shows that communication between nodes never uses
>>> messages > 4k which gets the network cards to top out on packages per
>>> second before they reach the top bits per second.
>>>
>>> Possibly related: I see exactly the same problem when using Basho Bench
>>> and the pb driver - I can never get the client to go over 2.5Gb/s. This is
>>> pushing me to suspect erlang vm or something as the problem.
>>>
>>> Anyone seen this before? Is there a magic "use bigger messages" flag
>>> hidden somewhere?
>>>
>>> Thanks,
>>>
>>> Chris
>>>
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>>
>>
>>
>> --
>> Sean Cribbs <s...@basho.com>
>> Software Engineer
>> Basho Technologies, Inc.
>> http://basho.com/
>>
>
>
--
Sean Cribbs <s...@basho.com>
Software Engineer
Basho Technologies, Inc.
http://basho.com/
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
