Re: riak handoffs stalled

2014-07-14 Thread Ciprian Manea
Hi Leonid, Lets try to increase the handoff_timeout and see if it can solve your problem. Could you please paste the below code in a $ riak attach riak_core_util:rpc_every_member_ann(application,set_env,[riak_core, handoff_timeout, 540],infinity). riak_core_util:rpc_every_member_ann(applicat

Re: riak handoffs stalled

2014-07-14 Thread Леонид Рябоштан
Hello, riak version is 1.1.4-1. We set transfer limit in config made it equal to 4. I don't think we have riak-admin transfer-limit or riak-admin cluster plan. The problem is that damn nodes can't pass partition between each other, probably because they're too big. Each 5k files(leveldb backend)

Re: riak handoffs stalled

2014-07-14 Thread Ciprian Manea
Hi Leonid, Which Riak version are you running? Have you committed* the cluster plan after issuing the cluster force-remove commands? What is the output of $ riak-admin transfer-limit, ran from one of your riak nodes? *Do not run this command yet if you have not done it already. Please run a r

riak handoffs stalled

2014-07-14 Thread Леонид Рябоштан
Hello, guys, It seems like we ran into emergency. I wonder if there can be any help on that. Everything that happened below was because we were trying to rebalace space used by nodes that we running out of space. Cluster is 7 machines now, member_status looks like: Attempting to restart script t