Thanks Georgio, I just noticed that we went off list so I’m copying riak-users back in. I always forget to reply-all.
Also realised I never asked, what version of Riak are you running (looks 1.4.something?) On 20 Dec 2013, at 10:01, Georgio Pandarez <georgiopanda...@gmail.com> wrote: > Hi Russell, > > > What about the drives? Actually, with a single Key you’re probably not > > touching the drives at all. > > Just a single SATA disk. I've only got two keys and am testing a single key > at a time. > > > So is there a single HTTP connection? Many? How many? Does it use a pool of > > connections that are kept alive, establishing an HTTP connection is pricey. > > Is all the load going to one node? Spread across all 5? > > I am using haproxy to load balance requests against the 5 servers. I am using > keepalive connections and have tested from 5 connections to 200 connections. > After a certain point increasing the connections only increases latency. > That makes sense, there is usually a sweetspot for number of connections. All your clients are contending for the same key. > Looking at the servers, riak is using most of the CPU, I'm not sure what it > is doing. > > running riak-admin top from one of the servers shows the below, I've tried > disabling riak_kv_stat but it still hangs around: You can’t disable stats in riak 1.4.x which is looks like your using (from the inclusion of side job in the etop output) > > Load: cpu 114 Memory: total 90204 binary > 672 > procs 670 processes 8103 code > 10535 > runq 1 atom 493 ets > 30233 > > Pid Name or Initial Func Time Reds Memory > MsgQ Current Function > ------------------------------------------------------------------------------------------------------------------------------- > <6210.577.0> proc_lib:init_p/5 '-' 7415450 34688 > 0 gen_fsm:loop/7 > <6210.2977.0> proc_lib:init_p/5 '-' 761452 2880 > 1 prim_file:drv_get_response/1 > <6210.304.0> riak_kv_stat_sj_1 '-' 371484 3856 > 0 gen_server:loop/6 > <6210.305.0> riak_kv_stat_sj_2 '-' 363239 3856 > 0 gen_server:loop/6 > <6210.1054.19> proc_lib:init_p/5 '-' 125656 34328 > 0 mochiweb_http:request/3 > <6210.753.19> proc_lib:init_p/5 '-' 125588 34328 > 0 mochiweb_http:request/3 > <6210.1041.19> proc_lib:init_p/5 '-' 125588 34328 > 0 mochiweb_http:request/3 > <6210.1052.19> proc_lib:init_p/5 '-' 125588 34328 > 0 mochiweb_http:request/3 > <6210.664.19> proc_lib:init_p/5 '-' 124041 21552 > 0 riak_client:wait_for_reqid/3 > <6210.694.19> proc_lib:init_p/5 '-' 124041 21552 > 0 riak_client:wait_for_reqid/3 It doesn’t look like Riak is doing a whole lot at this point. I’ll speak with my colleagues when they get online for some more guidance, as I’m not sure what numbers you should expect for a single key. Are you going to try with 100s, 1000s, millions of keys (which is far more common usage)? What is the use case that requires you get the same key thousands of times a second? Cheers Russell > > > > On Fri, Dec 20, 2013 at 8:51 PM, Russell Brown <russell.br...@me.com> wrote: > Hi Georgio, > > Thanks for getting back to me. > > On 20 Dec 2013, at 09:23, Georgio Pandarez <georgiopanda...@gmail.com> wrote: > > > Hi Russell, > > > > Thanks for your reply. > > > > > Are you getting 2400 reqs a second against a single key? > > > > Yes. > > Ah, OK. Well that is OKish for a single key, though I’ve seen better. > > > > > > What backend are you using? > > > > bitcask > > > > > What is the spec of the machines? > > > > dual core, 4gig, physical, 100mbit between the machines > > What about the drives? Actually, with a single Key you’re probably not > touching the drives at all. > > > > > > Is this perf figure against cs or riak? > > > > This perf figure is against riak. I was originally trying riak-cs and > > wasn't getting satisfactory performance, so I decided to try directly > > against riak. > > > > > Have you tried the PB driver? > > > > No > > Please do. It should be faster. > > > > > > What is the ring size? > > > > 64 > > > > > What “requests” are you doing? Are they serial in a loop using CURL or > > > parallel using basho_bench? > > > > GET requests using a http performance testing tool like httpress > > So is there a single HTTP connection? Many? How many? Does it use a pool of > connections that are kept alive, establishing an HTPP connection is pricey. > Is all the load going to one node? Spread across all 5? > > > > > > Something is wrong, because on my local 3 desktop cluster I can get well > > > over 5000ops/sec. > > > > Yes, it seems that way. I've been reading docs and watching video's for the > > past two weeks, so performance definitely isn't up to expectations. > > Well, usually you spread you’re load over many keys, so get greater > performance from parallelism, you also often have many clients, again, better > performance from parallelism. > > Let me know some more about the HTTP element as I’ve asked above, please. > > Cheers > > Russell > > > > > > > On Fri, Dec 20, 2013 at 7:44 PM, Russell Brown <russell.br...@me.com> wrote: > > Hi Georgio, > > > > With data that small I doubt there is a difference in perf. > > > > Can I get some more info, please? > > > > Are you getting 2400 reqs a second against a single key? What backend are > > you using? What is the spec of the machines? Are they real or on some > > cloud? Network? > > > > Is this perf figure against cs or riak? Have you tried the PB driver? What > > is the ring size? What “requests” are you doing? Are they serial in a loop > > using CURL or parallel using basho_bench? > > > > Something is wrong, because on my local 3 desktop cluster I can get well > > over 5000ops/sec. > > > > If you can provide more info then we can probably help you get performance > > closer to your expectations. > > > > Cheers > > > > Russell > > > > > > On 20 Dec 2013, at 08:33, Georgio Pandarez <georgiopanda...@gmail.com> > > wrote: > > > > > Hi Guys, > > > > > > I'm still evaluating riak & riak-cs. > > > > > > On my five node cluster, with riak only, I am only able to do 2400 > > > requests per second. I have two keys, one a one byte key, and another a > > > 4kb key, the performance doesn't change against either. > > > > > > Is this the best that I can expect to get from this, I don't consider > > > 2400 rps to be significantly high for 5 machines. I'm testing using the > > > http interface. > > > _______________________________________________ > > > riak-users mailing list > > > riak-users@lists.basho.com > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
