Hi Sargun,
>>1) I recommend you have a 5-node cluster: We'll add another node for sure. >> 2) What version of Riak are you using? 2.0.1 >> 3) What backend(s) are you using? leveldb >> 4) What's the size of your keyspace? Not sure what to answer here. >> 5) Are you actively rewriting keys, or writing keys to the cluster? Yes, plenty of other operations on this cluster for reading/writing etc. This particular key does not get written to anymore once its initially created. >> 6) Do you know how much I/O the cluster is currently doing? Very little at the moment (almost non-existant) as we haven't ported any of our clients onto this new system just yet - we ran a load test against our system and then 2 days later without any changes at all we started noticing this timeout issue. We turned on debug messages temporarily and have noticed this in the logs: 2014-12-29 11:51:44.957 [error] <0.14490.1> gen_fsm <0.14490.1> in state waiting_vnode_r terminated with reason: bad argument in call to erlang:hd([]) in riak_object:most_recent_content/1 line 228 2014-12-29 11:51:44.958 [error] <0.14490.1> CRASH REPORT Process <0.14490.1> with 0 neighbours exited with reason: bad argument in call to erlang:hd([]) in riak_object:most_recent_content/1 line 228 in gen_fsm:terminate/7 line 622 Thanks, Jason [image: photo] *Jason Ryan* VP Engineering Trustev Real Time, Online Identity Verification email: jason.r...@trustev.com skype: jason_j_ryan web: www.trustev.com Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland. On 29 December 2014 at 11:55, Sargun Dhillon <sar...@sargun.me> wrote: > Several things: > 1) I recommend you have a 5-node cluster: > http://basho.com/why-your-riak-cluster-should-have-at-least-five-nodes/ > 2) What version of Riak are you using? > 3) What backend(s) are you using? > 4) What's the size of your keyspace? > 5) Are you actively rewriting keys, or writing keys to the cluster? > 6) Do you know how much I/O the cluster is currently doing? > > On Mon, Dec 29, 2014 at 2:51 AM, Jason Ryan <jason.r...@trustev.com> > wrote: > > Hi, > > > > We are getting random timeouts from our application (>60seconds) when we > try > > to retrieve a key from our Riak cluster (4 nodes with a load balancer in > > front of them). Our application just uses the standard REST API to query > > Riak. > > > > We are pretty new to Riak - so would like to understand how best to debug > > this issue? Is there any good pointers on what to start with? This is our > > production cluster. > > > > Thanks, > > Jason > > > > > > This message is for the named person's use only. If you received this > > message in error, please immediately delete it and all copies and notify > the > > sender. You must not, directly or indirectly, use, disclose, distribute, > > print, or copy any part of this message if you are not the intended > > recipient. Any views expressed in this message are those of the > individual > > sender and not Trustev Ltd. Trustev is registered in Ireland No. 516425 > and > > trades from 2100 Cork Airport Business Park, Cork, Ireland. > > > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > -- This message is for the named person's use only. If you received this message in error, please immediately delete it and all copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender and not Trustev Ltd. Trustev is registered in Ireland No. 516425 and trades from 2100 Cork Airport Business Park, Cork, Ireland.
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
