Toby, Can you trying putting data with a second client simultaneously? When people have slow benchmarking, lots of times just using multiple worker/clients helps. Also, what client library are you using?
Also I meant to mention in my first reply, but Boundary http://boundary.com/worked wonders for us being able to see how much data was really moving around. They have a free trial as far as I know. It might be worth it to see if there are any obvious bottlenecks. Good luck, Jared On Mon, May 27, 2013 at 8:46 PM, Toby Corkindale < toby.corkind...@strategicdata.com.au> wrote: > On 28/05/13 01:41, Jared Morrow wrote: > >> Toby, >> >> If you write with multiple clients does it still stick to 9mb/s or >> does it increase? What is the network link between your client and >> the Riak CS cluster? On our internal CS cluster we were seeing >> around 2gb/s read+write at the network level so I know CS can take >> the speeds, so my gut thinks you single client might have a slow >> link. That is just a guess. >> > > The network links are all Ethernet, and appear to be functioning OK. > iperf reports: > bandwidth from client to loadbalancer: 2.20 gbit/sec > bandwidth from loadbalancer to a riak node: 941 mbit/sec > bandwidth from one riak node to another node: 942 mbit/s > > I've tested going direct from a client to a riak node rather than via the > loadbalancer, but it doesn't seem to make any difference. > > Having tested a bit further now, I'd guess that the problem lies with Riak > rather than Riak CS. > > I've noticed that if I try to push files directly into Riak, they go > fairly slowly too - around 10-20mbyte/sec. > > I've tried 3, 10 and 50 MB files, against bitcask, leveldb and even memory > backends, and in all cases I get fairly consistent transfer rates in that > range. (just using curl for testing here) > > I've tried reducing n_val to 1, there was a small but not significant > improvement. > > I'm a bit stumped.. However I do note that the log files seem to have a > lot of "monitor busy_dist_port" messages in them.. I'm wondering if that > might be related somehow? > > > T > > > On May 27, 2013, at 1:38 AM, Toby Corkindale >> <toby.corkindale@**strategicdata.com.au<toby.corkind...@strategicdata.com.au>> >> wrote: >> >> Hi, I seem to be getting significantly slower performance out of >>> Riak CS than I expect to see, and I wondered if I could get some >>> advice about where I might be going wrong. >>> >>> I have a cluster of six nodes, with fairly good sequential-write >>> disk performance, for eg: # dd if=/dev/zero of=test bs=1M count=500 >>> conv=fsync 524288000 bytes (524 MB) copied, 1.27205 s, 412 MB/s >>> >>> The riak partition is formatted as ext4, with data=writeback and >>> barriers disabled. >>> >>> So given n_val=3, and six nodes with ~400MB/s transfer rates, you'd >>> expect to see some pretty fast transfers, right? >>> >>> However the actual performance I'm getting is reliably a mere 9MB/s >>> for a single client writing to the cluster. (On a 500M transfer) >>> >>> >>> Is this kind of performance normal for this hardware? If not, could >>> you give me some pointers about what to check? >>> >>> Thanks, Toby >>> >>> >>> ______________________________**_________________ riak-users mailing >>> list riak-users@lists.basho.com >>> http://lists.basho.com/**mailman/listinfo/riak-users_**lists.basho.com<http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com> >>> >> >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
