On 28/05/13 13:11, Jared Morrow wrote:
Toby,
Can you trying putting data with a second client simultaneously? When
people have slow benchmarking, lots of times just using multiple
worker/clients helps. Also, what client library are you using?
Running up three S3 clients (on separate machines) simultaneously saw
them return 8, 8, 6 MB/sec. Interesting to note that the performance
hasn't dropped threefold, but still, it'd be really nice if an
individual transfer would run faster, given the performance of the
underlying hardware.
The nodes hardly get utilised during a write operation. I've uploading a
total of 1500M from the three nodes, and yet CPUs are 90% idle, and
there's no real disk activity going on, apart from a couple of times
when several hundred get flushed out over the course of a second.
I feel like something isn't quite right. What is the system *waiting*
for? There's plenty of CPU and IO to go around.
In the case of the current benchmarking, I'm using either s3cmd or curl.
Also I meant to mention in my first reply, but Boundary
http://boundary.com/ worked wonders for us being able to see how much
data was really moving around. They have a free trial as far as I know.
It might be worth it to see if there are any obvious bottlenecks.
Thanks, I'll have a look and see if the effort of setting it all up
looks worthwhile.
Cheers,
Toby
On Mon, May 27, 2013 at 8:46 PM, Toby Corkindale
<toby.corkind...@strategicdata.com.au
<mailto:toby.corkind...@strategicdata.com.au>> wrote:
On 28/05/13 01:41, Jared Morrow wrote:
Toby,
If you write with multiple clients does it still stick to 9mb/s or
does it increase? What is the network link between your client and
the Riak CS cluster? On our internal CS cluster we were seeing
around 2gb/s read+write at the network level so I know CS can take
the speeds, so my gut thinks you single client might have a slow
link. That is just a guess.
The network links are all Ethernet, and appear to be functioning OK.
iperf reports:
bandwidth from client to loadbalancer: 2.20 gbit/sec
bandwidth from loadbalancer to a riak node: 941 mbit/sec
bandwidth from one riak node to another node: 942 mbit/s
I've tested going direct from a client to a riak node rather than
via the loadbalancer, but it doesn't seem to make any difference.
Having tested a bit further now, I'd guess that the problem lies
with Riak rather than Riak CS.
I've noticed that if I try to push files directly into Riak, they go
fairly slowly too - around 10-20mbyte/sec.
I've tried 3, 10 and 50 MB files, against bitcask, leveldb and even
memory backends, and in all cases I get fairly consistent transfer
rates in that range. (just using curl for testing here)
I've tried reducing n_val to 1, there was a small but not
significant improvement.
I'm a bit stumped.. However I do note that the log files seem to
have a lot of "monitor busy_dist_port" messages in them.. I'm
wondering if that might be related somehow?
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
