Re: Riak crash on node restarts

2011-12-01 Thread Jeremy Raymond
I haven't had a chance to track down all the logs, ring state, etc and reproduce this issue, but I've updated my deploy script to just reload the mapped module while Riak is running rather than bouncing the nodes. I did however recently upgrade from Riak 1.0.0 to 1.0.2 and saw the same 3rd node

Re: Riak crash on node restarts

2011-11-18 Thread Jeremy Raymond
My db is only a few gigs. You are right it may be better to just load the updated mapred module without bouncing the node since we can just reload the individual module without bringing Riak down. -- Jeremy On Nov 18, 2011, at 5:06 PM, Leonid Riaboshtan wrote: Well, depends on your database siz

Re: Riak crash on node restarts

2011-11-18 Thread Leonid Riaboshtan
Well, depends on your database size really and on large variety of other things. Our database is like 40 GB of pure data(n_val 3 on most of data), and usually it takes 5-10 minutes for handoffs to complete on 256 vnodes ring. Handoff concurrency is set to 1 btw. I guess it's strange when handoffs g

Re: Riak crash on node restarts

2011-11-18 Thread Jeremy Raymond
Something else I tried to give the cluster more time to settle was to wait until riak-admin transfers reported no pending transfers between updating nodes. I've had cases where the transfers didn't complete within at least a couple of hours of waiting. What would be typical amount of time for pendi

Re: Riak crash on node restarts

2011-11-18 Thread Jeremy Raymond
Hello, I'll setup my deploy script to capture this information and send you the info off-list (probably sometime next week). -- Jeremy On 2011-11-15, at 1:16 PM, Jon Meredith wrote: > Hi Joel, > > That's not a message I'd expect to see on a clean restart. We'll need some > more information

Re: Riak crash on node restarts

2011-11-15 Thread Jon Meredith
Hi Joel, That's not a message I'd expect to see on a clean restart. We'll need some more information to diagnose it. Next time it crashes, could you provide the contents of your ring file (you can just grab the most recent one out of /var/lib/riak/ring - location may vary depending on your platf

Riak crash on node restarts

2011-11-15 Thread Jeremy Raymond
I'm using Riak 1.0.1 and I have a script that deploys updates to each of my 3 nodes to update the Erlang mapred modules. What I do is stop a node, deploy the new mapred modues, restart the node, wait for the riak_kv service to start, then move onto the next node. Sometimes when I do this one of the