We're seeing regular errors around indexes trying to be created - but they already exist - indexes are only ever created manually, and no new indexes have been created in quite a while. We have however added new nodes and removed other nodes due to some node failures.
We also have large object warnings, >5MB (generally around 7MB) - which we address each time we see them and mitigate the errors for the future... Other than that we don't see many other errors, so we're unsure why the cluster tends to fail. [image: photo] *Jason Ryan* VP Engineering Trustev Real Time, Online Identity Verification email: jason.r...@trustev.com skype: jason_j_ryan web: www.trustev.com Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland. On 4 March 2015 at 16:14, Sean Cribbs <s...@basho.com> wrote: > Are there any other significant error messages in the logs? I find it hard > to believe that that specific crash would make your cluster unusable. > > On Wed, Mar 4, 2015 at 10:07 AM, Jason Ryan <jason.r...@trustev.com> > wrote: > >> This is what we have on our haproxy right now: >> >> backend riak_rest_backend >> mode http >> option httplog >> balance roundrobin >> option httpchk GET /ping >> server EU-RIAK-5 10.0.4.12:8098 weight 1 maxconn 1024 check >> server EU-RIAK-6 10.0.4.13:8098 weight 1 maxconn 1024 check >> server EU-RIAK-7 10.0.4.14:8098 weight 1 maxconn 1024 check >> server EU-RIAK-8 10.0.4.15:8098 weight 1 maxconn 1024 check >> server EU-RIAK-9 10.0.4.16:8098 weight 1 maxconn 1024 check >> ᐧ >> >> >> >> [image: photo] >> *Jason Ryan* >> VP Engineering >> >> Trustev >> Real Time, Online Identity Verification >> >> email: jason.r...@trustev.com >> skype: jason_j_ryan >> web: www.trustev.com >> >> Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland. >> >> On 4 March 2015 at 15:58, Sean Cribbs <s...@basho.com> wrote: >> >>> We recently saw this on the PB interface as well and have mitigated it >>> [1], but we have not addressed it on HTTP. In the meantime, I would >>> recommend changing your health check to use a regular request, for example: >>> >>> option httpchk /ping >>> >>> [1] https://github.com/basho/riak_api/pull/54 >>> >>> On Wed, Mar 4, 2015 at 9:46 AM, Jason Ryan <jason.r...@trustev.com> >>> wrote: >>> >>>> Hi Sean - yes we are using HAProxy - and there are health checks. >>>> >>>> [image: photo] >>>> *Jason Ryan* >>>> VP Engineering >>>> >>>> Trustev >>>> Real Time, Online Identity Verification >>>> >>>> email: jason.r...@trustev.com >>>> skype: jason_j_ryan >>>> web: www.trustev.com >>>> >>>> Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland. >>>> >>>> On 4 March 2015 at 15:41, Sean Cribbs <s...@basho.com> wrote: >>>> >>>>> Are you using a load balancer in front of Riak, like HAProxy, with >>>>> health checks turned on? >>>>> >>>>> On Wed, Mar 4, 2015 at 7:24 AM, Jason Ryan <jason.r...@trustev.com> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> We're getting the following crash report messages in the log very >>>>>> regularly in the last day - it renders our Riak Cluster pretty much >>>>>> unusable - until we restart Riak on each node. All nodes report the same >>>>>> type of error. >>>>>> >>>>>> 2015-03-04 13:17:43.106 [error] <0.31063.9> CRASH REPORT Process >>>>>> <0.31063.9> with 0 neighbours crashed with reason: no function clause >>>>>> matching webmachine_request:peer_from_peername({error,enotconn}, >>>>>> {webmachine_request,{wm_reqstate,#Port<0.89869>,[],undefined,undefined,undefined,{wm_reqdata,'GET',...},...}}) >>>>>> line 150 >>>>>> >>>>>> >>>>>> Anyone seen this before, know what the cause might be? >>>>>> >>>>>> Thanks, >>>>>> Jason >>>>>> >>>>>> [image: photo] >>>>>> *Jason Ryan* >>>>>> VP Engineering >>>>>> >>>>>> Trustev >>>>>> Real Time, Online Identity Verification >>>>>> >>>>>> email: jason.r...@trustev.com >>>>>> skype: jason_j_ryan >>>>>> web: www.trustev.com >>>>>> >>>>>> Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland. >>>>>> >>>>>> This message is for the named person's use only. If you received this >>>>>> message in error, please immediately delete it and all copies and notify >>>>>> the sender. You must not, directly or indirectly, use, disclose, >>>>>> distribute, print, or copy any part of this message if you are not the >>>>>> intended recipient. Any views expressed in this message are those of the >>>>>> individual sender and not Trustev Ltd. Trustev is registered in >>>>>> Ireland No. 516425 and trades from 2100 Cork Airport Business Park, Cork, >>>>>> Ireland. >>>>>> >>>>>> _______________________________________________ >>>>>> riak-users mailing list >>>>>> riak-users@lists.basho.com >>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Sean Cribbs <s...@basho.com> >>>>> Sr. Software Engineer >>>>> Basho Technologies, Inc. >>>>> http://basho.com/ >>>>> >>>> >>>> >>>> This message is for the named person's use only. If you received this >>>> message in error, please immediately delete it and all copies and notify >>>> the sender. You must not, directly or indirectly, use, disclose, >>>> distribute, print, or copy any part of this message if you are not the >>>> intended recipient. Any views expressed in this message are those of the >>>> individual sender and not Trustev Ltd. Trustev is registered in >>>> Ireland No. 516425 and trades from 2100 Cork Airport Business Park, Cork, >>>> Ireland. >>>> >>> >>> >>> >>> -- >>> Sean Cribbs <s...@basho.com> >>> Sr. Software Engineer >>> Basho Technologies, Inc. >>> http://basho.com/ >>> >> >> >> This message is for the named person's use only. If you received this >> message in error, please immediately delete it and all copies and notify >> the sender. You must not, directly or indirectly, use, disclose, >> distribute, print, or copy any part of this message if you are not the >> intended recipient. Any views expressed in this message are those of the >> individual sender and not Trustev Ltd. Trustev is registered in Ireland >> No. 516425 and trades from 2100 Cork Airport Business Park, Cork, Ireland. >> > > > > -- > Sean Cribbs <s...@basho.com> > Sr. Software Engineer > Basho Technologies, Inc. > http://basho.com/ > -- This message is for the named person's use only. If you received this message in error, please immediately delete it and all copies and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Any views expressed in this message are those of the individual sender and not Trustev Ltd. Trustev is registered in Ireland No. 516425 and trades from 2100 Cork Airport Business Park, Cork, Ireland.
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
