Thanks Sean, I'll try the batch threshold and report back.
On Mon, Oct 28, 2013 at 10:26 AM, Sean Cribbs <s...@basho.com> wrote:
> Chris,
>
> I've spoken to our performance-tuning experts and unfortunately they say
> there's not too much we can do with 1.4 to eek out more bandwidth for
> handoff. However, the two settings you might try:
>
> {riak_core, [
> {handoff_batch_threshold, 1048576},
> {handoff_concurrency, 2} ]}
>
> handoff_batch_threshold is how big of a buffer it will collect before
> doing a TCP send. You could bump that up to, say, 4MB (4 * 1024 * 1024),
> but we have found it to be somewhat inconclusive in tests.
>
> The other option (which can be set at runtime using riak-admin) is
> handoff_concurrency, which is how many handoffs (incoming + outgoing) can
> be running at a time. Increasing this will also increase disk IO
> utilization, so tune it carefully.
>
> We are looking at a lot of ways to increase performance in Riak 2.0, but
> as many performance things go, the bottlenecks are not always what we first
> think. I'm sure our perf experts will be able to discuss this more as 2.0
> approaches.
>
> Sorry I can't provide a more satisfactory answer.
>
>
> On Mon, Oct 28, 2013 at 9:59 AM, Chris Read <chris.r...@gmail.com> wrote:
>
>> Yes, we have have jumbo frames on.
>>
>> Kernel and systems are in a good state. I've got the settings from the
>> Linux perf tuning guide on the riak docs. Testing with iperf(1) shows our
>> nodes can saturate the 10G interfaces without breaking a sweat - as long as
>> the packets are large enough...
>>
>> 1) http://iperf.sourceforge.net/
>>
>>
>> On Sat, Oct 26, 2013 at 11:36 AM, Sean Cribbs <s...@basho.com> wrote:
>>
>>> Chris,
>>>
>>> Are you using jumbo frames on your network? (MTU 9000 or w/e) That's
>>> just one aspect, but we've also found there are many other kernel/TCP-stack
>>> tunings that need to be done to get Riak to push the limits of the network.
>>>
>>>
>>> On Fri, Oct 25, 2013 at 9:10 PM, Chris Read <chris.r...@gmail.com>wrote:
>>>
>>>> Greetings all...
>>>>
>>>> Context: We're running Riak 1.4.2 on Ubuntu 12.04 using the Basho build
>>>> .deb package. Out object size varies between 1K and 512K
>>>>
>>>> I've been having some trouble trying to figure out why handoffs between
>>>> our nodes takes so long and never pushes the network card over 2.5 Gb/s.
>>>> Testing communication between nodes using iperf shows that the
>>>> configuration we have can sustain 10G.
>>>>
>>>> Further investigation shows that communication between nodes never uses
>>>> messages > 4k which gets the network cards to top out on packages per
>>>> second before they reach the top bits per second.
>>>>
>>>> Possibly related: I see exactly the same problem when using Basho Bench
>>>> and the pb driver - I can never get the client to go over 2.5Gb/s. This is
>>>> pushing me to suspect erlang vm or something as the problem.
>>>>
>>>> Anyone seen this before? Is there a magic "use bigger messages" flag
>>>> hidden somewhere?
>>>>
>>>> Thanks,
>>>>
>>>> Chris
>>>>
>>>> _______________________________________________
>>>> riak-users mailing list
>>>> riak-users@lists.basho.com
>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>>>
>>>
>>>
>>> --
>>> Sean Cribbs <s...@basho.com>
>>> Software Engineer
>>> Basho Technologies, Inc.
>>> http://basho.com/
>>>
>>
>>
>
>
> --
> Sean Cribbs <s...@basho.com>
> Software Engineer
> Basho Technologies, Inc.
> http://basho.com/
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
