On 29/05/13 06:12, Reid Draper wrote:
Hey Toby,
Another option is to explore some of the Riak CS PUT configuration parameters,
like the internal write concurrency and the buffer size. These can either be
changed by editing the app.config, or changing them at run-time in an Erlang
shell (via riak-cs attach). The two parameters are `put_concurrency` and
`put_buffer_factor`.
They both default to 1. The `put_concurrency` controls the number of threads
inside of Riak CS that are used to write blocks to Riak. The
`put_buffer_factor` controls the number of blocks that will be buffered
in-memory in Riak CS before it starts to slow down reading from the HTTP
client. I suggesting trying to raise these values to get higher single-client
throughput. If you wish to edit the app.config, add lines like in the `riak_cs`
section:
{put_concurrency, 8},
{put_buffer_factor, 16},
Ah, thanks -- that's interesting. Bumping up those up a bit did improve
things a bit -- I went for a conservative concurrency of 4,
buffer_factor of 8, and that took speeds on 500M files from 8-9mb/s to
12-13mb/sec.
Might play with other values when I next get time, but for now that's a
good cheap win.
In increasing these values, you might also find it useful to run a load
balancer between the Riak CS and Riak nodes, instead of having Riak CS just
communicate with the local Riak node. We intend to have this behavior built
into Riak CS in the future, but for now a load balancer will suffice.
Ah, ta. Can do.
Furthermore, please make sure you've looked through the Linux Tuning page for
Riak [1]. And as for +zdbbl, you can try even higher, like 128MB: +zdbbl 131072.
Thanks.
I'd been through there already, and applied things like filesystem and
scheduler options.
I haven't touched the networking sysctls since they came with a warning
that they should only be messed with if networking *was* the bottleneck.
Given what I've mentioned so far, do you think it's likely?
I guess it can't hurt to adjust them and see..
[1]
http://docs.basho.com/riak/1.3.1/cookbooks/Linux-Performance-Tuning/#Linux-Tuning
Thanks for your advice,
Toby
Reid
On May 27, 2013, at 11:50 PM, Toby Corkindale
<toby.corkind...@strategicdata.com.au> wrote:
On 28/05/13 13:11, Jared Morrow wrote:
Toby,
Can you trying putting data with a second client simultaneously? When
people have slow benchmarking, lots of times just using multiple
worker/clients helps. Also, what client library are you using?
Running up three S3 clients (on separate machines) simultaneously saw them
return 8, 8, 6 MB/sec. Interesting to note that the performance hasn't dropped
threefold, but still, it'd be really nice if an individual transfer would run
faster, given the performance of the underlying hardware.
The nodes hardly get utilised during a write operation. I've uploading a total
of 1500M from the three nodes, and yet CPUs are 90% idle, and there's no real
disk activity going on, apart from a couple of times when several hundred get
flushed out over the course of a second.
I feel like something isn't quite right. What is the system *waiting* for?
There's plenty of CPU and IO to go around.
In the case of the current benchmarking, I'm using either s3cmd or curl.
Also I meant to mention in my first reply, but Boundary
http://boundary.com/ worked wonders for us being able to see how much
data was really moving around. They have a free trial as far as I know.
It might be worth it to see if there are any obvious bottlenecks.
Thanks, I'll have a look and see if the effort of setting it all up looks
worthwhile.
Cheers,
Toby
On Mon, May 27, 2013 at 8:46 PM, Toby Corkindale
<toby.corkind...@strategicdata.com.au
<mailto:toby.corkind...@strategicdata.com.au>> wrote:
On 28/05/13 01:41, Jared Morrow wrote:
Toby,
If you write with multiple clients does it still stick to 9mb/s or
does it increase? What is the network link between your client and
the Riak CS cluster? On our internal CS cluster we were seeing
around 2gb/s read+write at the network level so I know CS can take
the speeds, so my gut thinks you single client might have a slow
link. That is just a guess.
The network links are all Ethernet, and appear to be functioning OK.
iperf reports:
bandwidth from client to loadbalancer: 2.20 gbit/sec
bandwidth from loadbalancer to a riak node: 941 mbit/sec
bandwidth from one riak node to another node: 942 mbit/s
I've tested going direct from a client to a riak node rather than
via the loadbalancer, but it doesn't seem to make any difference.
Having tested a bit further now, I'd guess that the problem lies
with Riak rather than Riak CS.
I've noticed that if I try to push files directly into Riak, they go
fairly slowly too - around 10-20mbyte/sec.
I've tried 3, 10 and 50 MB files, against bitcask, leveldb and even
memory backends, and in all cases I get fairly consistent transfer
rates in that range. (just using curl for testing here)
I've tried reducing n_val to 1, there was a small but not
significant improvement.
I'm a bit stumped.. However I do note that the log files seem to
have a lot of "monitor busy_dist_port" messages in them.. I'm
wondering if that might be related somehow?
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
