Re: Crash Error

2015-03-04 Thread Jason Ryan
Hi Sean - yes we are using HAProxy - and there are health checks. [image: photo] *Jason Ryan* VP Engineering Trustev Real Time, Online Identity Verification email: jason.r...@trustev.com skype: jason_j_ryan web: www.trustev.com Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland. On 4

Re: Crash Error

2015-03-04 Thread Jason Ryan
We're seeing regular errors around indexes trying to be created - but they already exist - indexes are only ever created manually, and no new indexes have been created in quite a while. We have however added new nodes and removed other nodes due to some node failures. We also have large object war

Re: Crash Error

2015-03-04 Thread Sean Cribbs
Are there any other significant error messages in the logs? I find it hard to believe that that specific crash would make your cluster unusable. On Wed, Mar 4, 2015 at 10:07 AM, Jason Ryan wrote: > This is what we have on our haproxy right now: > > backend riak_rest_backend > mode

Re: Crash Error

2015-03-04 Thread Jason Ryan
This is what we have on our haproxy right now: backend riak_rest_backend mode http option httplog balanceroundrobin option httpchk GET /ping server EU-RIAK-5 10.0.4.12:8098 weight 1 maxconn 1024 check

Re: Crash Error

2015-03-04 Thread Jason Ryan
We actually use /ping already for the health check [image: photo] *Jason Ryan* VP Engineering Trustev Real Time, Online Identity Verification email: jason.r...@trustev.com skype: jason_j_ryan web: www.trustev.com Trustev Ltd, 2100 Cork Airport Business Park, Cork, Ireland. On 4 March 2015 at 1

Re: Crash Error

2015-03-04 Thread Sean Cribbs
We recently saw this on the PB interface as well and have mitigated it [1], but we have not addressed it on HTTP. In the meantime, I would recommend changing your health check to use a regular request, for example: option httpchk /ping [1] https://github.com/basho/riak_api/pull/54 On Wed, Mar 4,

Re: Crash Error

2015-03-04 Thread Sean Cribbs
Are you using a load balancer in front of Riak, like HAProxy, with health checks turned on? On Wed, Mar 4, 2015 at 7:24 AM, Jason Ryan wrote: > Hi all, > > We're getting the following crash report messages in the log very > regularly in the last day - it renders our Riak Cluster pretty much > un