High cpu usage on store() , small number of keys
I'm starting to play with Riak, so far I am very impressed with the
product.
I want to use Riak as an highly available distributed cache.
I've compiled Erlang R1304 from source (--with-ssl, --enable-smp-support,
--enable-kernel-poll)
I've compiled riak-0.14.2 from source
I use 2 nodes, they are running on two RHEL5 box . (2.6.18.8.el5 #1 SMP).
They are clustered.
I use HAProxy to load balance the request on the two nodes.
I have a test bench that plays with 80 keys in one bucket (I use
riak_kv_cache_backend)
I use the java client (riak-client 0.14.1 fetched using a maven repo)
I have 10 clients that does something like this :
while(true) {
Fetch a key from the bucket
Sleep between 100 and 400ms
}
I have also 10 servers that updates data in the keys every 6 seconds (80
keys)
conceptually they do something like this :
while (true) {
Fetch a key from the bucket
Update the data
Store the key in the bucket
Sleep 6000ms
}
The number of clients have a little impact on the CPU usage of the nodes.
However when I start 10 servers, the CPU usage goes very high (I saw one of
the node going up to 200%)
Is there something I should check or do to lower the CPU usage on the nodes
when I write keys to Riak?
Thanks a lot!
-tony
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: High cpu usage on store() , small number of keys
Hi Ryan,
The cpu usage is mostly constant, on both nodes. I tried to play with my N
but it doesn't make a significant difference.
The 10 writes are happening almost concurrently.
Is there a way to show the impact of the conflict resolutions (stats or
something like it)?
thanks!
-tony
On Mon, Aug 8, 2011 at 10:29 PM, Ryan Zezeski wrote:
> Tony,
>
> Is it a constant 200% usage? On both nodes in the cluster? My first
> thought, since your working with a small key-space, would be conflict
> resolution or vclock explosion but you're spacing the writes out by 6s which
> seems like more than enough time. Are all 10 writes happening at the same
> time concurrently?
>
> I would set your N=2 since you only have two machines, or add a 3rd node.
>
> Perhaps there is some weird behavior in the cache backend, but I would be
> surprised if that was it.
>
> -Ryan
>
> On Wed, Aug 3, 2011 at 11:31 AM, Tony Bussieres wrote:
>
>> I'm starting to play with Riak, so far I am very impressed with the
>> product.
>>
>> I want to use Riak as an highly available distributed cache.
>>
>> I've compiled Erlang R1304 from source (--with-ssl, --enable-smp-support,
>> --enable-kernel-poll)
>> I've compiled riak-0.14.2 from source
>> I use 2 nodes, they are running on two RHEL5 box . (2.6.18.8.el5 #1 SMP).
>> They are clustered.
>> I use HAProxy to load balance the request on the two nodes.
>>
>> I have a test bench that plays with 80 keys in one bucket (I use
>> riak_kv_cache_backend)
>> I use the java client (riak-client 0.14.1 fetched using a maven repo)
>>
>>
>> I have 10 clients that does something like this :
>> while(true) {
>>Fetch a key from the bucket
>>Sleep between 100 and 400ms
>> }
>>
>> I have also 10 servers that updates data in the keys every 6 seconds (80
>> keys)
>> conceptually they do something like this :
>> while (true) {
>>Fetch a key from the bucket
>>Update the data
>>Store the key in the bucket
>>Sleep 6000ms
>> }
>>
>> The number of clients have a little impact on the CPU usage of the nodes.
>> However when I start 10 servers, the CPU usage goes very high (I saw one
>> of the node going up to 200%)
>>
>> Is there something I should check or do to lower the CPU usage on the
>> nodes when I write keys to Riak?
>>
>> Thanks a lot!
>>
>> -tony
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
riak_kv_cache_backend config questions
Hi all, Could someone explain with an example what's the difference between : *riak_kv_cache_backend_max_ttl and **riak_kv_cache_backend_ttl Does the value of riak_kv_backend_max_ttl is also in seconds? Here's a snippet from the documentation found at http://wiki.basho.com/Configuration-Files.html * - *riak_kv_cache_backend* A backend that behaves as an LRU-with-timed-expiry cache - *riak_kv_cache_backend_memory* Maximum amount of memory to allocate, in megabytes (default: “100”) - *riak_kv_cache_backend_ttl* Amount by which to extend an object’s expiry lease on each access, in seconds (default: “600”) - *riak_kv_cache_backend_max_ttl* Maximum allowed lease time (default: “3600”) Thanks! -tony ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak_kv_cache_backend config questions
Thanks Kelly! So, if I understood well. with a config like this : riak_kv_cache_backend_ttl 60 *riak_kv_cache_backend_max_ttl 300 *If I put a key at time 0, and access it will disappear after 60 second. If I put a key at time 0, and access it at time 50, it will disappear at time 110 If I put a key at time 0, and access it every 15 seconds, it will disappear after 300 seconds. I'm I right? Regards, -tony On Tue, Aug 23, 2011 at 10:22 AM, Kelly McLaughlin wrote: > Tony, > > riak_kv_cache_backend_ttl is the amount of time to extend an object's lease > or lifespan when accessed whereas *riak_kv_cache_backend_max_ttl is the > amount of time after which no further extensions should be granted. * > > Kelly > > > On Aug 23, 2011, at 7:49 AM, Tony Bussieres wrote: > > Hi all, > > Could someone explain with an example what's the difference between : > > *riak_kv_cache_backend_max_ttl > and > **riak_kv_cache_backend_ttl > > Does the value of riak_kv_backend_max_ttl is also in seconds? > > Here's a snippet from the documentation found at > http://wiki.basho.com/Configuration-Files.html > * > >- *riak_kv_cache_backend* >A backend that behaves as an LRU-with-timed-expiry cache > - *riak_kv_cache_backend_memory* > Maximum amount of memory to allocate, in megabytes (default: “100”) > - *riak_kv_cache_backend_ttl* > Amount by which to extend an object’s expiry lease on each access, > in seconds (default: “600”) > - *riak_kv_cache_backend_max_ttl* > Maximum allowed lease time (default: “3600”) > > Thanks! > > -tony > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Keys aren't deleted
Hi,
We have a cluster of two nodes.
We add keys to a bucket :
curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
http://riak/store/jdTestBucket/key1?w=1
curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
http://riak/store/jdTestBucket/key2?w=1
curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
http://riak/store/jdTestBucket/key3?w=1
curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
http://riak/store/jdTestBucket/key4?w=1
We shut down one node.
We delete the keys:
curl -v -X DELETE http://riak/store/jdTestBucket/key1?w=1
curl -v -X DELETE http://riak/store/jdTestBucket/key2?w=1
curl -v -X DELETE http://riak/store/jdTestBucket/key3?w=1
curl -v -X DELETE http://riak/store/jdTestBucket/key4?w=1
We list the keys :
curl -i http://riak/store/jdTestBucket?keys=true\&props=false
All keys are listed.
Is it normal?
Regards,
-tony
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Keys aren't deleted
Hi Andrew,
Can we automatically, after restarting a node, read-repair all entries using
the command line?
Or we have to read all keys individually so they become consistent?
-tony
On Wed, Aug 31, 2011 at 5:27 PM, Andrew Thompson wrote:
> On Wed, Aug 31, 2011 at 05:11:16PM -0400, Tony Bussieres wrote:
> > Hi,
> >
> > We have a cluster of two nodes.
> >
> > We add keys to a bucket :
> > curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
> > http://riak/store/jdTestBucket/key1?w=1
> > curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
> > http://riak/store/jdTestBucket/key2?w=1
> > curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
> > http://riak/store/jdTestBucket/key3?w=1
> > curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
> > http://riak/store/jdTestBucket/key4?w=1
> >
> > We shut down one node.
> >
> > We delete the keys:
> > curl -v -X DELETE http://riak/store/jdTestBucket/key1?w=1
> > curl -v -X DELETE http://riak/store/jdTestBucket/key2?w=1
> > curl -v -X DELETE http://riak/store/jdTestBucket/key3?w=1
> > curl -v -X DELETE http://riak/store/jdTestBucket/key4?w=1
> >
> >
> > We list the keys :
> > curl -i http://riak/store/jdTestBucket?keys=true\&props=false
> >
> > All keys are listed.
> >
> > Is it normal?
> >
>
> Yes, keys are only deleted when all the values are tombstones and all
> the nodes are up. In theory bringing the other node back up and
> re-reading the keys will read-repair the keys on the node that was down
> to be tombstones and then hopefully they'll be removed.
>
> However, deletes are flaky on 0.14. Things should be a little better in
> 1.0.
>
> Andrew
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak crash on 0.14.2 riak_kv_stat terminating
Hi Sean,
I have the same issue as mentioned here :
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004639.html
It happens few time every day, the node doesn't crash, I'm not sure if I
have to be worried about this or not.
I have plenty of memory available on this server.
Here's the output of erlang.log
I'm having difficulty to decipher this stack trace any howto available ?
:)
Thanks
-tony
= ALIVE Thu Sep 1 04:22:44 EDT 2011
= Thu Sep 1 04:36:44 EDT 2011
^M
=ERROR REPORT 1-Sep-2011::04:36:44 ===^M
** Generic server riak_kv_stat terminating ^M
** Last message in was {'$gen_cast',{update,vnode_get,63482071004}}^M
** When Server state == {state,^M
{spiral,63482071064,^M
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,^M
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,^M
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},^M
{spiral,63482071064,^M
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,^M
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,^M
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},^M
3054432,1513015,822300,400094,^M
{slide,63482056664,60,60,^M
"/tmp/riak/slide-data/28906/1314.851864.220785", ^H^M
{file_descriptor,prim_file,^M
{#Port<0.1369133>,25}},^M
63482071003},^M
{slide,63482056664,60,60,^M
"/tmp/riak/slide-data/28906/1314.851864.221558", ^H^M
{file_descriptor,prim_file,^M
{#Port<0.1369134>,11}},^M
63482071003},^M
{spiral,63482071064,^M
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,^M
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,^M
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},^M
0,0,^M
{spiral,63482071064,^M
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,^M
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,^M
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},^M
45,0}^M
** Reason for termination == ^M
** {badarg,[{erlang,hd,[[]]},^M
{spiraltime,incr,3},^M
{riak_kv_stat,spiral_incr,3},^M
{riak_kv_stat,handle_cast,2},^M
{gen_server2,handle_msg,7},^M
{proc_lib,init_p_do_apply,3}]}^M
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Keys aren't deleted
Hi,
I hope this script will help someone. (attached)
It reads all buckets and for each bucket it gets the keys and read all the
entries.
Could be potentially useful after the restart of a node.
-tony
On Thu, Sep 1, 2011 at 9:22 AM, Sean Cribbs wrote:
> You will need to do so manually. Riak currently has no active anti-entropy
> (expect the EDS product which has a kind of it in the long-haul
> replication).
>
>
> On Thu, Sep 1, 2011 at 9:20 AM, Tony Bussieres wrote:
>
>> Hi Andrew,
>>
>> Can we automatically, after restarting a node, read-repair all entries
>> using the command line?
>> Or we have to read all keys individually so they become consistent?
>>
>> -tony
>>
>>
>> On Wed, Aug 31, 2011 at 5:27 PM, Andrew Thompson wrote:
>>
>>> On Wed, Aug 31, 2011 at 05:11:16PM -0400, Tony Bussieres wrote:
>>> > Hi,
>>> >
>>> > We have a cluster of two nodes.
>>> >
>>> > We add keys to a bucket :
>>> > curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
>>> > http://riak/store/jdTestBucket/key1?w=1
>>> > curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
>>> > http://riak/store/jdTestBucket/key2?w=1
>>> > curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
>>> > http://riak/store/jdTestBucket/key3?w=1
>>> > curl -v -X PUT -d '{"prop":"val"}' -H "Content-Type: application/json"
>>> > http://riak/store/jdTestBucket/key4?w=1
>>> >
>>> > We shut down one node.
>>> >
>>> > We delete the keys:
>>> > curl -v -X DELETE http://riak/store/jdTestBucket/key1?w=1
>>> > curl -v -X DELETE http://riak/store/jdTestBucket/key2?w=1
>>> > curl -v -X DELETE http://riak/store/jdTestBucket/key3?w=1
>>> > curl -v -X DELETE http://riak/store/jdTestBucket/key4?w=1
>>> >
>>> >
>>> > We list the keys :
>>> > curl -i http://riak/store/jdTestBucket?keys=true\&props=false
>>> >
>>> > All keys are listed.
>>> >
>>> > Is it normal?
>>> >
>>>
>>> Yes, keys are only deleted when all the values are tombstones and all
>>> the nodes are up. In theory bringing the other node back up and
>>> re-reading the keys will read-repair the keys on the node that was down
>>> to be tombstones and then hopefully they'll be removed.
>>>
>>> However, deletes are flaky on 0.14. Things should be a little better in
>>> 1.0.
>>>
>>> Andrew
>>>
>>> ___
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>
>>
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> --
> Sean Cribbs
> Developer Advocate
> Basho Technologies, Inc.
> http://www.basho.com/
>
>
read_repair.sh
Description: Bourne shell script
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
