Re: Random timeouts on Riak

2014-12-30 Thread Jason Ryan
It was enabled by default - we didn't change this value when creating the bucket. The only properties on the bucket that we changed were allow_mult and last_write_wins. Further update - yesterday we also noticed in the logs that one of our keys, its JSON had grown to 17MB after some load tests, we

Re: Random timeouts on Riak

2014-12-30 Thread Russell Brown
On 30 Dec 2014, at 11:16, Jason Ryan wrote: > Hi Russell, > > Bucket Properties are: > > young_vclock: 20 > w: quorum > small_vclock: 50 > rw: quorum > r: quorum > pw: 0 > precommit: [] > pr: 0 > postcommit: [] > old_vclock: 86400 > notfound_ok: true > n_val: 3 > linkfun: {modfun,riak_kv_wm_li

Re: Random timeouts on Riak

2014-12-30 Thread Jason Ryan
Hi Russell, Bucket Properties are: young_vclock: 20 w: quorum small_vclock: 50 rw: quorum r: quorum pw: 0 precommit: [] pr: 0 postcommit: [] old_vclock: 86400 notfound_ok: true n_val: 3 linkfun: {modfun,riak_kv_wm_link_walker,mapreduce_linkfun} last_write_wins: true dw: quorum dvv_enabled: true c

Re: Random timeouts on Riak

2014-12-30 Thread Russell Brown
Please can you let me know if you’re using a typed or default bucket? Please can you let me know what you’re DVV setting is? On 29 Dec 2014, at 12:27, Jason Ryan wrote: > Also we noticed these warnings - should we change the environment to make > these on each node? > > 2014-12-29 11:32:14.56

Re: Random timeouts on Riak

2014-12-29 Thread Russell Brown
Alrighty. On 29 Dec 2014, at 12:29, Russell Brown wrote: > Hi Jason, > > I opened https://github.com/basho/riak_kv/issues/1069. Feel free to add any > information to it that you think is pertinent. > > On 29 Dec 2014, at 12:26, Jason Ryan wrote: > >> No - those settings were set during the

Re: Random timeouts on Riak

2014-12-29 Thread Russell Brown
Hi Jason, I opened https://github.com/basho/riak_kv/issues/1069. Feel free to add any information to it that you think is pertinent. On 29 Dec 2014, at 12:26, Jason Ryan wrote: > No - those settings were set during the setup of the cluster. > > >> Anyway you can get each of the N values from

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
Also we noticed these warnings - should we change the environment to make these on each node? 2014-12-29 11:32:14.564 [warning] <0.339.0> riak_kv_env: sysctl net.core.rmem_max is 124928, should be at least 8388608) 2014-12-29 11:32:14.564 [warning] <0.339.0> riak_kv_env: sysctl net.core.netdev_max

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
No - those settings were set during the setup of the cluster. >> Anyway you can get each of the N values from their backends separately so I can see there content lists? The N value is 3 - or is it a particular command you want us to run on each node? Thanks, Jason On 29 December 2014 at 12:19

Re: Random timeouts on Riak

2014-12-29 Thread Russell Brown
On 29 Dec 2014, at 12:09, Jason Ryan wrote: > All types/buckets we use are set to allow_mult: false - last_write_wins:true Did you change to this setting after these keys were written? Looks like a bug in Riak, so I’m going to open a ticket: hd([]) should never be called in reconcile. But any

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
All types/buckets we use are set to allow_mult: false - last_write_wins:true [image: photo] *Jason Ryan* VP Engineering Trustev Real Time, Online Identity Verification email: jason.r...@trustev.com skype: jason_j_ryan web: www.trustev.com Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ir

Re: Random timeouts on Riak

2014-12-29 Thread Sargun Dhillon
The bucket (type) that you're working with -- what are your allow_mult, and last_write_wins settings? On Mon, Dec 29, 2014 at 4:05 AM, Jason Ryan wrote: > It seems to move between 4 keys in particular, these keys are actually empty > at the moment (i.e. an empty JSON document). > > CPU utilizatio

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
It seems to move between 4 keys in particular, these keys are actually empty at the moment (i.e. an empty JSON document). CPU utilization is close to zero. Can't see anything in particular, bar the error message I just posted before. Jason On 29 December 2014 at 11:58, Ciprian Manea wrote: >

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
Hi Sargun, >>1) I recommend you have a 5-node cluster: We'll add another node for sure. >> 2) What version of Riak are you using? 2.0.1 >> 3) What backend(s) are you using? leveldb >> 4) What's the size of your keyspace? Not sure what to answer here. >> 5) Are you actively rewriting keys

Re: Random timeouts on Riak

2014-12-29 Thread Ciprian Manea
Hi Jason, Are these random timeouts happening for only one key, or is common for more? What is the CPU utilisation in the cluster when you're experience these timeouts? Can you spot anything peculiar in your server's $ dmesg outputs? Any I/O errors there? Regards, Ciprian On Mon, Dec 29, 2014

Re: Random timeouts on Riak

2014-12-29 Thread Sargun Dhillon
Several things: 1) I recommend you have a 5-node cluster: http://basho.com/why-your-riak-cluster-should-have-at-least-five-nodes/ 2) What version of Riak are you using? 3) What backend(s) are you using? 4) What's the size of your keyspace? 5) Are you actively rewriting keys, or writing keys to the

Random timeouts on Riak

2014-12-29 Thread Jason Ryan
Hi, We are getting random timeouts from our application (>60seconds) when we try to retrieve a key from our Riak cluster (4 nodes with a load balancer in front of them). Our application just uses the standard REST API to query Riak. We are pretty new to Riak - so would like to understand how best