I feel the need to point out that with just a key or two you're
probably only talking to 3/5 of the machines. for a single key adding
extra nodes won't make it faster. (As I understand it)
- Peace
Dave
On Fri, Dec 20, 2013 at 3:35 AM, Russell Brown <russell.br...@me.com> wrote:
>
> On 20 Dec 2013, at 10:18, Georgio Pandarez <georgiopanda...@gmail.com> wrote:
>
>> Hi Russell,
>>
>> Thanks for the prompt response.
>>
>> > Also realised I never asked, what version of Riak are you running
>>
>> 1.4.2
>>
>> > It doesn’t look like Riak is doing a whole lot at this point
>>
>> So is it normal for the cpu to spin?
>
> Again, I’ll check with colleagues later, I don’t see that behaviour locally,
> I have a suspicion this is the stats gathering process running in the
> background. Though you can’t turn it off, you can give it a really long
> interval and see if that helps, You can do this by adding
>
> {stat_cache_ttl, $SomeTimeInSeconds}
>
> to the riak_core section in your app.config
>
>>
>> > . I’ll speak with my colleagues when they get online for some more
>> > guidance, as I’m not sure what numbers you should expect for a single key.
>> > Are you going to try with 100s, 1000s, millions of keys (which is far more
>> > common usage)? What is the use case that requires you get the same key
>> > thousands of times a second?
>>
>> Yes, I will be trying with many keys, I just wanted to see if riak is
>> suitable for this application.
>>
>> What kind of rps should I be expecting on a typical 5 node cluster of a read
>> heavy workload? I know there are so many variables, but should each node be
>> generating 1000 rps?
>
> Yes, you should be able to achieve 1000s ops/sec per node. We have a tuning
> section in our docs that is worth a read
> http://docs.basho.com/riak/1.4.0/cookbooks/Linux-Performance-Tuning/
>
> Hopefully I can get some more concrete answers for you later in the day.
>
> Cheers
>
> Russell
>
>>
>>
>>
>>
>> On Fri, Dec 20, 2013 at 9:08 PM, Russell Brown <russell.br...@me.com> wrote:
>> Thanks Georgio,
>>
>> I just noticed that we went off list so I’m copying riak-users back in. I
>> always forget to reply-all.
>>
>> Also realised I never asked, what version of Riak are you running (looks
>> 1.4.something?)
>>
>> On 20 Dec 2013, at 10:01, Georgio Pandarez <georgiopanda...@gmail.com> wrote:
>>
>> > Hi Russell,
>> >
>> > > What about the drives? Actually, with a single Key you’re probably not
>> > > touching the drives at all.
>> >
>> > Just a single SATA disk. I've only got two keys and am testing a single
>> > key at a time.
>> >
>> > > So is there a single HTTP connection? Many? How many? Does it use a pool
>> > > of connections that are kept alive, establishing an HTTP connection is
>> > > pricey. Is all the load going to one node? Spread across all 5?
>> >
>> > I am using haproxy to load balance requests against the 5 servers. I am
>> > using keepalive connections and have tested from 5 connections to 200
>> > connections. After a certain point increasing the connections only
>> > increases latency.
>> >
>>
>> That makes sense, there is usually a sweetspot for number of connections.
>> All your clients are contending for the same key.
>>
>> > Looking at the servers, riak is using most of the CPU, I'm not sure what
>> > it is doing.
>> >
>> > running riak-admin top from one of the servers shows the below, I've tried
>> > disabling riak_kv_stat but it still hangs around:
>>
>> You can’t disable stats in riak 1.4.x which is looks like your using (from
>> the inclusion of side job in the etop output)
>>
>> >
>> > Load: cpu 114 Memory: total 90204 binary
>> > 672
>> > procs 670 processes 8103 code
>> > 10535
>> > runq 1 atom 493 ets
>> > 30233
>> >
>> > Pid Name or Initial Func Time Reds
>> > Memory MsgQ Current Function
>> > -------------------------------------------------------------------------------------------------------------------------------
>> > <6210.577.0> proc_lib:init_p/5 '-' 7415450
>> > 34688 0 gen_fsm:loop/7
>> > <6210.2977.0> proc_lib:init_p/5 '-' 761452
>> > 2880 1 prim_file:drv_get_response/1
>> > <6210.304.0> riak_kv_stat_sj_1 '-' 371484
>> > 3856 0 gen_server:loop/6
>> > <6210.305.0> riak_kv_stat_sj_2 '-' 363239
>> > 3856 0 gen_server:loop/6
>> > <6210.1054.19> proc_lib:init_p/5 '-' 125656
>> > 34328 0 mochiweb_http:request/3
>> > <6210.753.19> proc_lib:init_p/5 '-' 125588
>> > 34328 0 mochiweb_http:request/3
>> > <6210.1041.19> proc_lib:init_p/5 '-' 125588
>> > 34328 0 mochiweb_http:request/3
>> > <6210.1052.19> proc_lib:init_p/5 '-' 125588
>> > 34328 0 mochiweb_http:request/3
>> > <6210.664.19> proc_lib:init_p/5 '-' 124041
>> > 21552 0 riak_client:wait_for_reqid/3
>> > <6210.694.19> proc_lib:init_p/5 '-' 124041
>> > 21552 0 riak_client:wait_for_reqid/3
>>
>> It doesn’t look like Riak is doing a whole lot at this point. I’ll speak
>> with my colleagues when they get online for some more guidance, as I’m not
>> sure what numbers you should expect for a single key. Are you going to try
>> with 100s, 1000s, millions of keys (which is far more common usage)? What is
>> the use case that requires you get the same key thousands of times a second?
>>
>> Cheers
>>
>> Russell
>>
>> >
>> >
>> >
>> > On Fri, Dec 20, 2013 at 8:51 PM, Russell Brown <russell.br...@me.com>
>> > wrote:
>> > Hi Georgio,
>> >
>> > Thanks for getting back to me.
>> >
>> > On 20 Dec 2013, at 09:23, Georgio Pandarez <georgiopanda...@gmail.com>
>> > wrote:
>> >
>> > > Hi Russell,
>> > >
>> > > Thanks for your reply.
>> > >
>> > > > Are you getting 2400 reqs a second against a single key?
>> > >
>> > > Yes.
>> >
>> > Ah, OK. Well that is OKish for a single key, though I’ve seen better.
>> >
>> > >
>> > > > What backend are you using?
>> > >
>> > > bitcask
>> > >
>> > > > What is the spec of the machines?
>> > >
>> > > dual core, 4gig, physical, 100mbit between the machines
>> >
>> > What about the drives? Actually, with a single Key you’re probably not
>> > touching the drives at all.
>> >
>> > >
>> > > > Is this perf figure against cs or riak?
>> > >
>> > > This perf figure is against riak. I was originally trying riak-cs and
>> > > wasn't getting satisfactory performance, so I decided to try directly
>> > > against riak.
>> > >
>> > > > Have you tried the PB driver?
>> > >
>> > > No
>> >
>> > Please do. It should be faster.
>> >
>> > >
>> > > > What is the ring size?
>> > >
>> > > 64
>> > >
>> > > > What “requests” are you doing? Are they serial in a loop using CURL or
>> > > > parallel using basho_bench?
>> > >
>> > > GET requests using a http performance testing tool like httpress
>> >
>> > So is there a single HTTP connection? Many? How many? Does it use a pool
>> > of connections that are kept alive, establishing an HTPP connection is
>> > pricey. Is all the load going to one node? Spread across all 5?
>> >
>> > >
>> > > > Something is wrong, because on my local 3 desktop cluster I can get
>> > > > well over 5000ops/sec.
>> > >
>> > > Yes, it seems that way. I've been reading docs and watching video's for
>> > > the past two weeks, so performance definitely isn't up to expectations.
>> >
>> > Well, usually you spread you’re load over many keys, so get greater
>> > performance from parallelism, you also often have many clients, again,
>> > better performance from parallelism.
>> >
>> > Let me know some more about the HTTP element as I’ve asked above, please.
>> >
>> > Cheers
>> >
>> > Russell
>> >
>> > >
>> > >
>> > > On Fri, Dec 20, 2013 at 7:44 PM, Russell Brown <russell.br...@me.com>
>> > > wrote:
>> > > Hi Georgio,
>> > >
>> > > With data that small I doubt there is a difference in perf.
>> > >
>> > > Can I get some more info, please?
>> > >
>> > > Are you getting 2400 reqs a second against a single key? What backend
>> > > are you using? What is the spec of the machines? Are they real or on
>> > > some cloud? Network?
>> > >
>> > > Is this perf figure against cs or riak? Have you tried the PB driver?
>> > > What is the ring size? What “requests” are you doing? Are they serial in
>> > > a loop using CURL or parallel using basho_bench?
>> > >
>> > > Something is wrong, because on my local 3 desktop cluster I can get well
>> > > over 5000ops/sec.
>> > >
>> > > If you can provide more info then we can probably help you get
>> > > performance closer to your expectations.
>> > >
>> > > Cheers
>> > >
>> > > Russell
>> > >
>> > >
>> > > On 20 Dec 2013, at 08:33, Georgio Pandarez <georgiopanda...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Guys,
>> > > >
>> > > > I'm still evaluating riak & riak-cs.
>> > > >
>> > > > On my five node cluster, with riak only, I am only able to do 2400
>> > > > requests per second. I have two keys, one a one byte key, and another
>> > > > a 4kb key, the performance doesn't change against either.
>> > > >
>> > > > Is this the best that I can expect to get from this, I don't consider
>> > > > 2400 rps to be significantly high for 5 machines. I'm testing using
>> > > > the http interface.
>> > > > _______________________________________________
>> > > > riak-users mailing list
>> > > > riak-users@lists.basho.com
>> > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> > >
>> > >
>> >
>> >
>>
>>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
