Hey Toby,
Another option is to explore some of the Riak CS PUT configuration parameters,
like the internal write concurrency and the buffer size. These can either be
changed by editing the app.config, or changing them at run-time in an Erlang
shell (via riak-cs attach). The two parameters are `put_concurrency` and
`put_buffer_factor`.
They both default to 1. The `put_concurrency` controls the number of threads
inside of Riak CS that are used to write blocks to Riak. The
`put_buffer_factor` controls the number of blocks that will be buffered
in-memory in Riak CS before it starts to slow down reading from the HTTP
client. I suggesting trying to raise these values to get higher single-client
throughput. If you wish to edit the app.config, add lines like in the `riak_cs`
section:
{put_concurrency, 8},
{put_buffer_factor, 16},
Or, if you wish to use `riak-cs attach` (note though that this will not
withstand a restart):
application:set_env(riak_cs, put_concurrency, 8).
application:set_env(riak_cs, put_buffer_factor, 16).
In increasing these values, you might also find it useful to run a load
balancer between the Riak CS and Riak nodes, instead of having Riak CS just
communicate with the local Riak node. We intend to have this behavior built
into Riak CS in the future, but for now a load balancer will suffice.
Furthermore, please make sure you've looked through the Linux Tuning page for
Riak [1]. And as for +zdbbl, you can try even higher, like 128MB: +zdbbl 131072.
[1]
http://docs.basho.com/riak/1.3.1/cookbooks/Linux-Performance-Tuning/#Linux-Tuning
Reid
On May 27, 2013, at 11:50 PM, Toby Corkindale
<toby.corkind...@strategicdata.com.au> wrote:
> On 28/05/13 13:11, Jared Morrow wrote:
>> Toby,
>>
>> Can you trying putting data with a second client simultaneously? When
>> people have slow benchmarking, lots of times just using multiple
>> worker/clients helps. Also, what client library are you using?
>
> Running up three S3 clients (on separate machines) simultaneously saw them
> return 8, 8, 6 MB/sec. Interesting to note that the performance hasn't
> dropped threefold, but still, it'd be really nice if an individual transfer
> would run faster, given the performance of the underlying hardware.
>
> The nodes hardly get utilised during a write operation. I've uploading a
> total of 1500M from the three nodes, and yet CPUs are 90% idle, and there's
> no real disk activity going on, apart from a couple of times when several
> hundred get flushed out over the course of a second.
>
> I feel like something isn't quite right. What is the system *waiting* for?
> There's plenty of CPU and IO to go around.
>
>
> In the case of the current benchmarking, I'm using either s3cmd or curl.
>
>
>
>> Also I meant to mention in my first reply, but Boundary
>> http://boundary.com/ worked wonders for us being able to see how much
>> data was really moving around. They have a free trial as far as I know.
>> It might be worth it to see if there are any obvious bottlenecks.
>
> Thanks, I'll have a look and see if the effort of setting it all up looks
> worthwhile.
>
> Cheers,
> Toby
>
>
>> On Mon, May 27, 2013 at 8:46 PM, Toby Corkindale
>> <toby.corkind...@strategicdata.com.au
>> <mailto:toby.corkind...@strategicdata.com.au>> wrote:
>>
>> On 28/05/13 01:41, Jared Morrow wrote:
>>
>> Toby,
>>
>> If you write with multiple clients does it still stick to 9mb/s or
>> does it increase? What is the network link between your client and
>> the Riak CS cluster? On our internal CS cluster we were seeing
>> around 2gb/s read+write at the network level so I know CS can take
>> the speeds, so my gut thinks you single client might have a slow
>> link. That is just a guess.
>>
>>
>> The network links are all Ethernet, and appear to be functioning OK.
>> iperf reports:
>> bandwidth from client to loadbalancer: 2.20 gbit/sec
>> bandwidth from loadbalancer to a riak node: 941 mbit/sec
>> bandwidth from one riak node to another node: 942 mbit/s
>>
>> I've tested going direct from a client to a riak node rather than
>> via the loadbalancer, but it doesn't seem to make any difference.
>>
>> Having tested a bit further now, I'd guess that the problem lies
>> with Riak rather than Riak CS.
>>
>> I've noticed that if I try to push files directly into Riak, they go
>> fairly slowly too - around 10-20mbyte/sec.
>>
>> I've tried 3, 10 and 50 MB files, against bitcask, leveldb and even
>> memory backends, and in all cases I get fairly consistent transfer
>> rates in that range. (just using curl for testing here)
>>
>> I've tried reducing n_val to 1, there was a small but not
>> significant improvement.
>>
>> I'm a bit stumped.. However I do note that the log files seem to
>> have a lot of "monitor busy_dist_port" messages in them.. I'm
>> wondering if that might be related somehow?
>>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
