Re: Weird RIAK behavior

2014-12-29 Thread Claudio Cesar Sanchez Tejeda
Hi, The clocks weren't synchronized because the servers don't have direct access to internet or a NTP server. They can use a http proxy, so I installed and configured htpdate on them in order to maintain their clocks synchronized. The issue was resolved by synchronizing the clocks. Thanks! On T

Re: Random timeouts on Riak

2014-12-29 Thread Russell Brown
Alrighty. On 29 Dec 2014, at 12:29, Russell Brown wrote: > Hi Jason, > > I opened https://github.com/basho/riak_kv/issues/1069. Feel free to add any > information to it that you think is pertinent. > > On 29 Dec 2014, at 12:26, Jason Ryan wrote: > >> No - those settings were set during the

Re: datatypes, mapreduce & 2i

2014-12-29 Thread Sean Cribbs
Hi Cezary, Datatypes like sets do not create 2I entries. We hope to resolve this limitation in a more holistic fashion in the near future. In the meantime, you can use them with Search if you like. On Sun, Dec 28, 2014 at 8:30 PM, Cezary Kosko wrote: > Hi there, > > I've got 2 questions about r

Re: Random timeouts on Riak

2014-12-29 Thread Russell Brown
Hi Jason, I opened https://github.com/basho/riak_kv/issues/1069. Feel free to add any information to it that you think is pertinent. On 29 Dec 2014, at 12:26, Jason Ryan wrote: > No - those settings were set during the setup of the cluster. > > >> Anyway you can get each of the N values from

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
Also we noticed these warnings - should we change the environment to make these on each node? 2014-12-29 11:32:14.564 [warning] <0.339.0> riak_kv_env: sysctl net.core.rmem_max is 124928, should be at least 8388608) 2014-12-29 11:32:14.564 [warning] <0.339.0> riak_kv_env: sysctl net.core.netdev_max

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
No - those settings were set during the setup of the cluster. >> Anyway you can get each of the N values from their backends separately so I can see there content lists? The N value is 3 - or is it a particular command you want us to run on each node? Thanks, Jason On 29 December 2014 at 12:19

Re: Random timeouts on Riak

2014-12-29 Thread Russell Brown
On 29 Dec 2014, at 12:09, Jason Ryan wrote: > All types/buckets we use are set to allow_mult: false - last_write_wins:true Did you change to this setting after these keys were written? Looks like a bug in Riak, so I’m going to open a ticket: hd([]) should never be called in reconcile. But any

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
All types/buckets we use are set to allow_mult: false - last_write_wins:true [image: photo] *Jason Ryan* VP Engineering Trustev Real Time, Online Identity Verification email: jason.r...@trustev.com skype: jason_j_ryan web: www.trustev.com Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ir

Re: Random timeouts on Riak

2014-12-29 Thread Sargun Dhillon
The bucket (type) that you're working with -- what are your allow_mult, and last_write_wins settings? On Mon, Dec 29, 2014 at 4:05 AM, Jason Ryan wrote: > It seems to move between 4 keys in particular, these keys are actually empty > at the moment (i.e. an empty JSON document). > > CPU utilizatio

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
It seems to move between 4 keys in particular, these keys are actually empty at the moment (i.e. an empty JSON document). CPU utilization is close to zero. Can't see anything in particular, bar the error message I just posted before. Jason On 29 December 2014 at 11:58, Ciprian Manea wrote: >

Re: Random timeouts on Riak

2014-12-29 Thread Jason Ryan
Hi Sargun, >>1) I recommend you have a 5-node cluster: We'll add another node for sure. >> 2) What version of Riak are you using? 2.0.1 >> 3) What backend(s) are you using? leveldb >> 4) What's the size of your keyspace? Not sure what to answer here. >> 5) Are you actively rewriting keys

Re: Random timeouts on Riak

2014-12-29 Thread Ciprian Manea
Hi Jason, Are these random timeouts happening for only one key, or is common for more? What is the CPU utilisation in the cluster when you're experience these timeouts? Can you spot anything peculiar in your server's $ dmesg outputs? Any I/O errors there? Regards, Ciprian On Mon, Dec 29, 2014

Re: Random timeouts on Riak

2014-12-29 Thread Sargun Dhillon
Several things: 1) I recommend you have a 5-node cluster: http://basho.com/why-your-riak-cluster-should-have-at-least-five-nodes/ 2) What version of Riak are you using? 3) What backend(s) are you using? 4) What's the size of your keyspace? 5) Are you actively rewriting keys, or writing keys to the

Random timeouts on Riak

2014-12-29 Thread Jason Ryan
Hi, We are getting random timeouts from our application (>60seconds) when we try to retrieve a key from our Riak cluster (4 nodes with a load balancer in front of them). Our application just uses the standard REST API to query Riak. We are pretty new to Riak - so would like to understand how best