Re: invalid hintfiles after upgrde

2014-01-21 Thread Jan Zahradka - MALL.cz
Hello everybody, I finally forced the bitcask merge this way. It looks ok ;o) Partitions = ["433883298582611803841718934712646521460354973696","1347321821914426127719021955160323408745312813056"]. [bitcask:merge(string:concat("/var/lib/riak/bitcask/",P)) || P <- Partitions]. Thanks for help

Re: invalid hintfiles after upgrde

2014-01-21 Thread Jan Zahradka - MALL.cz
Hello, Thank You very much. You saved my nerves ;o) One more question. Do I have to wait? Or is it possible to force the Bitcask merge? Jan On 01/18/2014 02:16 AM, Charlie Voiselle wrote: Invalid hint files will be rebuilt automatically during the Bitcask merge process. I believe you are

Re: invalid hintfiles after upgrde

2014-01-17 Thread Charlie Voiselle
Invalid hint files will be rebuilt automatically during the Bitcask merge process. I believe you are seeing a one-time consequence of switching to signed hint files during the upgrade. Not to worry, they will be regenerated through normal merge activity. Regards, -cv On Jan 17, 2014 6:56 AM, "

invalid hintfiles after upgrde

2014-01-17 Thread Jan Zahradka - MALL.cz
Hi everybody, I have a 3 node Riak 1.1.1 cluster running on Centos 6.3. I wanted to upgrade it to the most recent version and I was told that first I have to upgrade to version 1.3.2 and then I can upgrade to 1.4.X version. So I followed the instructions in Rolling upgrades guide (stop, upgra

Re: Invalid hintfiles

2013-09-29 Thread Toby Corkindale
On 19/09/13 12:02, Seth Thomas wrote: Toby, Although I haven't traced a root cause just yet I've had several reports in IRC of odd behavior when using nginx as a load balancer for Riak CS. In almost every case, removing nginx from the equation and switching to haproxy or pointing directly at the

Re: Invalid hintfiles

2013-09-18 Thread Andrew Stone
Good to hear. :) On Wed, Sep 18, 2013 at 11:52 PM, Toby Corkindale < toby.corkind...@strategicdata.com.au> wrote: > Hi Andrew, > Thanks for that -- so far things are looking stable after making that > change. > > -T > > > On 19/09/13 13:27, Andrew Stone wrote: > >> Hi Toby, >> >> Can you try rai

Re: Invalid hintfiles

2013-09-18 Thread Toby Corkindale
Hi Andrew, Thanks for that -- so far things are looking stable after making that change. -T On 19/09/13 13:27, Andrew Stone wrote: Hi Toby, Can you try raising the pb_backlog to 128 in your riak app.config on each node. It's likely those disconnect errors are left over from the stampede of c

Re: Invalid hintfiles

2013-09-18 Thread Andrew Stone
Hi Toby, Can you try raising the pb_backlog to 128 in your riak app.config on each node. It's likely those disconnect errors are left over from the stampede of connections from the CS connection pool on startup. For one reason or another the resets don't come through and the hanging disconnected s

Re: Invalid hintfiles

2013-09-18 Thread Toby Corkindale
On 19/09/13 11:17, Luke Bakken wrote: The "error: disconnected" message is a good clue. If you can provide log files that may point to the cause. See below for logs from riak and riak cs, for a roughly five minute window during which I'd fiddled with the load balancer and DNS to try and isola

Re: Invalid hintfiles

2013-09-18 Thread Seth Thomas
Toby,Although I haven't traced a root cause just yet I've had several reports in IRC of odd behavior when using nginx as a load balancer for Riak CS. In almost every case, removing nginx from the equation and switching to haproxy or pointing directly at the nodes in the application has resolved iss

Re: Invalid hintfiles

2013-09-18 Thread Toby Corkindale
On 19/09/13 11:17, Luke Bakken wrote: Hi Toby, Invalid hint files won't cause Riak to fail requests - there must have been something else happening. Hint files are used by Riak to speed start time when loading a large key set. You mentioned a "load-balancer pool" - are you using something like

Re: Invalid hintfiles (was: Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls)

2013-09-18 Thread Luke Bakken
Hi Toby, Invalid hint files won't cause Riak to fail requests - there must have been something else happening. Hint files are used by Riak to speed start time when loading a large key set. You mentioned a "load-balancer pool" - are you using something like HAProxy to load-balance requests to your

Invalid hintfiles (was: Riak CS 1.4.1 bug? InvalidAccessKeyId reported for deletion calls)

2013-09-18 Thread Toby Corkindale
I found one Riak server was reporting a lot of errors like [error] <0.808.0> Hintfile '/var/lib/riak/bitcask/68507889249886074290797726533575766546371837952/3.bitcask.hint' invalid And the Riak CS logs contained a lot of messages about being unable to retrieve s3 user details because "error: