Hi
I have recently added two extra nodes to the now seven node Riak
cluster. The rebalancing following the expansion worked fine, except
for two partitions which seem to not being able to go through. Running
"riak-admin ring-status" shows the following:
============================== Ownership Handoff ==============================
Owner: riak@10.0.0.96
Next Owner: riak@10.0.0.93
Index: 239777612374601260017792042867515182912301432832
Waiting on: []
Complete: [riak_kv_vnode,riak_pipe_vnode]
Index: 696496874040508421956443553091353626554780352512
Waiting on: []
Complete: [riak_kv_vnode,riak_pipe_vnode]
-------------------------------------------------------------------------------
I can see from the log file on the source node (10.0.0.96) that it has
made numerous attempt to transfer the partitions, but it ends up
failing all the time. Here's an except of the log file showing the
lines from when the transfer attempt ends up failing:
2013-11-18 12:29:03.694 [error] emulator Error in process <0.5745.8>
on node 'riak@10.0.0.96' with exit value:
{badarg,[{erlang,binary_to_term,[<<29942
bytes>>],[]},{mi_segment,iterate_all_bytes,2,[{file,"src/mi_segment.erl"},{line,167}]},{mi_server,'-group_iterator/2-fun-1-',2,[{file,"src/mi_server.erl"},{line,725}]},{mi_server,'-group_iterator/2-fun-0-'...
2013-11-18 12:29:03.885 [error] <0.3269.0>@mi_server:handle_info:524
lookup/range failure:
{badarg,[{erlang,binary_to_term,[<<131,109,0,0,244,240,108,109,102,97,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,
111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,1
11,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,11
1,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111
,111,111,111,111,111,111,111,111,111,111,111,111,111,111,...>>],...},...]}
2013-11-18 12:29:03.889 [error]
<0.30353.0>@merge_index_backend:async_fold_fun:116 failed to iterate
the index with reason
{badarg,[{erlang,binary_to_term,[<<131,109,0,0,244,240,108,109,102,97,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,
111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,111,...>>],..
.},...]}
and partial acc
{{ho_acc,1,{error,closed},#Fun<riak_core_handoff_sender.15.9980475>,riak_search_vnode,<0.3268.0>,#Port<0.133923>,{696496874040508421956443553091353626554780352512,696496874040508421956443553091353626554780352512},{ho_stats,{1384,775537,708230},undefined,249,1050473},gen_tcp,246,1053542,[<<1,131,104,4,109,0,0,0,8,109,111,114,101,111,118,101,114,109,0,0,0,13,99,97,110,111,110,105,99,97,108,68,97,116,101,109,0,0,0,14,50,48,49,51,49,48,48,55,49,50,49,49,53,52,108,0,0,0,57,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,102,98,56,55,98,102,50,51,55,51,98,51,98,56,56,100,52,48,51,52,51,50,100,97,97,55,51,99,55,101,98,99,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,154,61,194,52,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,102,51,101,54,55,54,57,56,99,55,52,49,102,57,56,50,101,50,102,48,55,54,101,57,57,101,97,56,100,100,102,57,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,137,181,162,60,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,
48,55,49,50,49,49,53,52,35,101,98,57,51,102,52,55,48,53,52,55,52,50,56,98,49,49,50,101,57,98,50,98,49,52,51,52,57,52,52,98,50,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,31,143,101,69,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,101,55,49,52,57,49,53,52,49,102,48,55,102,102,97,99,51,48,98,51,52,51,56,52,49,98,56,50,54,55,56,48,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,214,30,30,54,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,101,53,56,53,97,52,48,49,98,55,97,52,56,57,56,48,53,55,54,53,101,99,98,98,98,98,101,52,97,101,48,100,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,41,65,120,56,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,53,52,35,100,101,56,99,99,99,51,56,53,56,52,48,53,97,51,49,57,49,48,97,51,55,57,57,98,49,54,52,53,100,55,50,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,222,186,251,66,243,234,4,104,3,109,0,0,0,47,50,48,49,51,49,48,48,55,49,50,49,49,...>>,...],...},...}
2013-11-18 12:29:03.889 [error]
<0.15384.7>@riak_core_handoff_sender:start_fold:269 ownership_transfer
transfer of riak_search_vnode from 'riak@10.0.0.96'
696496874040508421956443553091353626554780352512 to 'riak@10.0.0.93'
696496874040508421956443553091353626554780352512 failed because of
error:{badrecord,ho_acc}
[{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,193}]}]
I've got an idea that it me be caused by corrupt data, but the
question is then how I can get this corrected, enabling the transfer
to complete. Any suggestions?
We are using Riak 1.4.2 with LevelDB as the backend on Ubuntu 12.04.
Riak was installed through the apt.basho.com repository. Let me know
if you need any more information.
--
Jeppe Fihl Toustrup
Operations Engineer
Falcon Social
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
