On 20 Dec 2013, at 10:18, Georgio Pandarez <georgiopanda...@gmail.com> wrote:
> Hi Russell,
>
> Thanks for the prompt response.
>
> > Also realised I never asked, what version of Riak are you running
>
> 1.4.2
>
> > It doesn’t look like Riak is doing a whole lot at this point
>
> So is it normal for the cpu to spin?
Again, I’ll check with colleagues later, I don’t see that behaviour locally, I
have a suspicion this is the stats gathering process running in the background.
Though you can’t turn it off, you can give it a really long interval and see if
that helps, You can do this by adding
{stat_cache_ttl, $SomeTimeInSeconds}
to the riak_core section in your app.config
>
> > . I’ll speak with my colleagues when they get online for some more
> > guidance, as I’m not sure what numbers you should expect for a single key.
> > Are you going to try with 100s, 1000s, millions of keys (which is far more
> > common usage)? What is the use case that requires you get the same key
> > thousands of times a second?
>
> Yes, I will be trying with many keys, I just wanted to see if riak is
> suitable for this application.
>
> What kind of rps should I be expecting on a typical 5 node cluster of a read
> heavy workload? I know there are so many variables, but should each node be
> generating 1000 rps?
Yes, you should be able to achieve 1000s ops/sec per node. We have a tuning
section in our docs that is worth a read
http://docs.basho.com/riak/1.4.0/cookbooks/Linux-Performance-Tuning/
Hopefully I can get some more concrete answers for you later in the day.
Cheers
Russell
>
>
>
>
> On Fri, Dec 20, 2013 at 9:08 PM, Russell Brown <russell.br...@me.com> wrote:
> Thanks Georgio,
>
> I just noticed that we went off list so I’m copying riak-users back in. I
> always forget to reply-all.
>
> Also realised I never asked, what version of Riak are you running (looks
> 1.4.something?)
>
> On 20 Dec 2013, at 10:01, Georgio Pandarez <georgiopanda...@gmail.com> wrote:
>
> > Hi Russell,
> >
> > > What about the drives? Actually, with a single Key you’re probably not
> > > touching the drives at all.
> >
> > Just a single SATA disk. I've only got two keys and am testing a single key
> > at a time.
> >
> > > So is there a single HTTP connection? Many? How many? Does it use a pool
> > > of connections that are kept alive, establishing an HTTP connection is
> > > pricey. Is all the load going to one node? Spread across all 5?
> >
> > I am using haproxy to load balance requests against the 5 servers. I am
> > using keepalive connections and have tested from 5 connections to 200
> > connections. After a certain point increasing the connections only
> > increases latency.
> >
>
> That makes sense, there is usually a sweetspot for number of connections. All
> your clients are contending for the same key.
>
> > Looking at the servers, riak is using most of the CPU, I'm not sure what it
> > is doing.
> >
> > running riak-admin top from one of the servers shows the below, I've tried
> > disabling riak_kv_stat but it still hangs around:
>
> You can’t disable stats in riak 1.4.x which is looks like your using (from
> the inclusion of side job in the etop output)
>
> >
> > Load: cpu 114 Memory: total 90204 binary
> > 672
> > procs 670 processes 8103 code
> > 10535
> > runq 1 atom 493 ets
> > 30233
> >
> > Pid Name or Initial Func Time Reds Memory
> > MsgQ Current Function
> > -------------------------------------------------------------------------------------------------------------------------------
> > <6210.577.0> proc_lib:init_p/5 '-' 7415450 34688
> > 0 gen_fsm:loop/7
> > <6210.2977.0> proc_lib:init_p/5 '-' 761452 2880
> > 1 prim_file:drv_get_response/1
> > <6210.304.0> riak_kv_stat_sj_1 '-' 371484 3856
> > 0 gen_server:loop/6
> > <6210.305.0> riak_kv_stat_sj_2 '-' 363239 3856
> > 0 gen_server:loop/6
> > <6210.1054.19> proc_lib:init_p/5 '-' 125656 34328
> > 0 mochiweb_http:request/3
> > <6210.753.19> proc_lib:init_p/5 '-' 125588 34328
> > 0 mochiweb_http:request/3
> > <6210.1041.19> proc_lib:init_p/5 '-' 125588 34328
> > 0 mochiweb_http:request/3
> > <6210.1052.19> proc_lib:init_p/5 '-' 125588 34328
> > 0 mochiweb_http:request/3
> > <6210.664.19> proc_lib:init_p/5 '-' 124041 21552
> > 0 riak_client:wait_for_reqid/3
> > <6210.694.19> proc_lib:init_p/5 '-' 124041 21552
> > 0 riak_client:wait_for_reqid/3
>
> It doesn’t look like Riak is doing a whole lot at this point. I’ll speak with
> my colleagues when they get online for some more guidance, as I’m not sure
> what numbers you should expect for a single key. Are you going to try with
> 100s, 1000s, millions of keys (which is far more common usage)? What is the
> use case that requires you get the same key thousands of times a second?
>
> Cheers
>
> Russell
>
> >
> >
> >
> > On Fri, Dec 20, 2013 at 8:51 PM, Russell Brown <russell.br...@me.com> wrote:
> > Hi Georgio,
> >
> > Thanks for getting back to me.
> >
> > On 20 Dec 2013, at 09:23, Georgio Pandarez <georgiopanda...@gmail.com>
> > wrote:
> >
> > > Hi Russell,
> > >
> > > Thanks for your reply.
> > >
> > > > Are you getting 2400 reqs a second against a single key?
> > >
> > > Yes.
> >
> > Ah, OK. Well that is OKish for a single key, though I’ve seen better.
> >
> > >
> > > > What backend are you using?
> > >
> > > bitcask
> > >
> > > > What is the spec of the machines?
> > >
> > > dual core, 4gig, physical, 100mbit between the machines
> >
> > What about the drives? Actually, with a single Key you’re probably not
> > touching the drives at all.
> >
> > >
> > > > Is this perf figure against cs or riak?
> > >
> > > This perf figure is against riak. I was originally trying riak-cs and
> > > wasn't getting satisfactory performance, so I decided to try directly
> > > against riak.
> > >
> > > > Have you tried the PB driver?
> > >
> > > No
> >
> > Please do. It should be faster.
> >
> > >
> > > > What is the ring size?
> > >
> > > 64
> > >
> > > > What “requests” are you doing? Are they serial in a loop using CURL or
> > > > parallel using basho_bench?
> > >
> > > GET requests using a http performance testing tool like httpress
> >
> > So is there a single HTTP connection? Many? How many? Does it use a pool of
> > connections that are kept alive, establishing an HTPP connection is pricey.
> > Is all the load going to one node? Spread across all 5?
> >
> > >
> > > > Something is wrong, because on my local 3 desktop cluster I can get
> > > > well over 5000ops/sec.
> > >
> > > Yes, it seems that way. I've been reading docs and watching video's for
> > > the past two weeks, so performance definitely isn't up to expectations.
> >
> > Well, usually you spread you’re load over many keys, so get greater
> > performance from parallelism, you also often have many clients, again,
> > better performance from parallelism.
> >
> > Let me know some more about the HTTP element as I’ve asked above, please.
> >
> > Cheers
> >
> > Russell
> >
> > >
> > >
> > > On Fri, Dec 20, 2013 at 7:44 PM, Russell Brown <russell.br...@me.com>
> > > wrote:
> > > Hi Georgio,
> > >
> > > With data that small I doubt there is a difference in perf.
> > >
> > > Can I get some more info, please?
> > >
> > > Are you getting 2400 reqs a second against a single key? What backend
> > > are you using? What is the spec of the machines? Are they real or on some
> > > cloud? Network?
> > >
> > > Is this perf figure against cs or riak? Have you tried the PB driver?
> > > What is the ring size? What “requests” are you doing? Are they serial in
> > > a loop using CURL or parallel using basho_bench?
> > >
> > > Something is wrong, because on my local 3 desktop cluster I can get well
> > > over 5000ops/sec.
> > >
> > > If you can provide more info then we can probably help you get
> > > performance closer to your expectations.
> > >
> > > Cheers
> > >
> > > Russell
> > >
> > >
> > > On 20 Dec 2013, at 08:33, Georgio Pandarez <georgiopanda...@gmail.com>
> > > wrote:
> > >
> > > > Hi Guys,
> > > >
> > > > I'm still evaluating riak & riak-cs.
> > > >
> > > > On my five node cluster, with riak only, I am only able to do 2400
> > > > requests per second. I have two keys, one a one byte key, and another a
> > > > 4kb key, the performance doesn't change against either.
> > > >
> > > > Is this the best that I can expect to get from this, I don't consider
> > > > 2400 rps to be significantly high for 5 machines. I'm testing using the
> > > > http interface.
> > > > _______________________________________________
> > > > riak-users mailing list
> > > > riak-users@lists.basho.com
> > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > >
> > >
> >
> >
>
>
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
