On Monday 02 December 2013 18:10:32 Ryan Zezeski wrote:
> Ivan,
>
> What makes you think the index is damaged?
Only the mention of "badrec" which I would assume meant a damanged record.
> From what I can see this is a
> bug in the code assuming that a #ho_acc record is always returned but in
> fact an error tuple could also be returned. That error tuple could be
> caused by index corruption but there is nothing in your email to say for
> sure. Do you see any other errors in the log?
Yes:
[root@ip-10-196-50-172 riak]# tail error.log
2013-12-03 14:29:45.167 [error] <0.1833.0>@mi_server:handle_info:524
lookup/range failure:
{{nocatch,{error,iterate2,timeout_waiting_for_continue}},[{mi_server,iterate2,5,[{file,"src/mi_server.erl"},{line,682}]}]}
2013-12-03 14:30:06.132 [error]
<0.1909.0>@merge_index_backend:async_fold_fun:116 failed to iterate the index
with reason
{{nocatch,{error,iterate2,timeout_waiting_for_continue}},[{mi_server,iterate2,5,[{file,"src/mi_server.erl"},{line,682}]}]}
and partial acc
{{ho_acc,16,ok,#Fun<riak_core_handoff_sender.15.9980475>,riak_search_vnode,<0.1832.0>,#Port<0.616405>,{502391187832497878132516661246222288006726811648,502391187832497878132516661246222288006726811648},{ho_stats,{1386,80925,839128},undefined,4637,1071941},gen_tcp,27777,43222372,[<<1,131,104,4,109,0,0,0,5,117,115,97,103,101,109,0,0,0,30,111,114,105,103,105,110,97,108,95,100,97,116,97,95,99,111,111,114,100,105,110,97,116,111,114,95,116,105,109,101,109,0,0,0,20,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,51,49,90,108,0,0,0,2,104,3,109,0,0,0,25,55,48,52,52,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,51,49,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,5,51,187,167,205,231,4,104,3,109,0,0,0,25,54,49,49,50,59
,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,51,49,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,155,20,127,167,205,231,4,106>>,<<1,131,104,4,109,0,0,0,5,117,115,97,103,101,109,0,0,0,30,111,114,105,103,105,110,97,108,95,100,97,116,97,95,99,111,111,114,100,105,110,97,116,111,114,95,116,105,109,101,109,0,0,0,20,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,57,90,108,0,0,0,2,104,3,109,0,0,0,25,55,48,48,53,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,57,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,151,149,213,169,205,231,4,104,3,109,0,0,0,25,54,57,52,52,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,57,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,198,232,168,169,205,231,4,106>>,<<1,131,104,4,109,0,0,0,5,117,115,97,103,101,109,0,0,0,30,111,114,105,103,105,110,97,108,95,100,97,116,97,95,99,111,111,114,100,105,110,97,116,111,114,95,116,105,109,101,109,0,0,0,20,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,48,90,108,0
,0,0,2,104,3,109,0,0,0,25,54,57,56,55,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,48,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,121,143,181,165,205,231,4,104,3,109,0,0,0,25,54,50,48,57,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,48,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,68,222,165,165,205,231,4,106>>,<<1,131,104,4,109,0,0,0,5,117,115,97,103,101,109,0,0,0,30,111,114,105,103,105,110,97,108,95,100,97,116,97,95,99,111,111,114,100,105,110,97,116,111,114,95,116,105,109,101,109,0,0,0,20,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,11,104,3,109,0,0,0,25,55,48,51,48,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,184,197,81,149,205,231,4,104,3,109,0,0,0,25,55,48,48,57,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,161,67,223,145,205,231,4,104,3,109,0,0,0,25,55,48,48,53,59,50,48,49,
51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,225,221,223,145,205,231,4,104,3,109,0,0,0,25,54,57,54,57,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,77,179,222,145,205,231,4,104,3,109,0,0,0,25,54,57,54,55,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,45,134,224,145,205,231,4,104,3,109,0,0,0,25,54,57,53,51,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,11,106,208,145,205,231,4,104,3,109,0,0,0,25,54,57,52,52,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,73,21,225,145,205,231,4,104,3,109,0,0,0,25,54,57,52,50,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,224,54,222,145,205,231,4,104,3,109,0,0,
0,25,54,57,50,50,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,...>>,...],...},...}
2013-12-03 14:30:06.132 [error]
<0.18468.37>@riak_core_handoff_sender:start_fold:269 hinted_handoff transfer of
riak_search_vnode from 'riak@10.196.50.172'
502391187832497878132516661246222288006726811648 to 'riak@10.197.0.55'
502391187832497878132516661246222288006726811648 failed because of
error:{badrecord,ho_acc}
[{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,193}]}]
2013-12-03 14:35:32.546 [error] emulator Error in process <0.20961.37> on node
'riak@10.196.50.172' with exit value:
{{nocatch,{error,iterate2,timeout_waiting_for_continue}},[{mi_server,iterate2,5,[{file,"src/mi_server.erl"},{line,682}]}]}
2013-12-03 14:35:32.546 [error] <0.1833.0>@mi_server:handle_info:524
lookup/range failure:
{{nocatch,{error,iterate2,timeout_waiting_for_continue}},[{mi_server,iterate2,5,[{file,"src/mi_server.erl"},{line,682}]}]}
2013-12-03 14:35:45.454 [error]
<0.1908.0>@merge_index_backend:async_fold_fun:116 failed to iterate the index
with reason
{{nocatch,{error,iterate2,timeout_waiting_for_continue}},[{mi_server,iterate2,5,[{file,"src/mi_server.erl"},{line,682}]}]}
and partial acc
{{ho_acc,16,ok,#Fun<riak_core_handoff_sender.15.9980475>,riak_search_vnode,<0.1832.0>,#Port<0.617641>,{502391187832497878132516661246222288006726811648,502391187832497878132516661246222288006726811648},{ho_stats,{1386,81273,907256},undefined,4637,1071941},gen_tcp,27777,43222372,[<<1,131,104,4,109,0,0,0,5,117,115,97,103,101,109,0,0,0,30,111,114,105,103,105,110,97,108,95,100,97,116,97,95,99,111,111,114,100,105,110,97,116,111,114,95,116,105,109,101,109,0,0,0,20,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,51,49,90,108,0,0,0,2,104,3,109,0,0,0,25,55,48,52,52,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,51,49,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,5,51,187,167,205,231,4,104,3,109,0,0,0,25,54,49,49,50,59
,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,51,49,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,155,20,127,167,205,231,4,106>>,<<1,131,104,4,109,0,0,0,5,117,115,97,103,101,109,0,0,0,30,111,114,105,103,105,110,97,108,95,100,97,116,97,95,99,111,111,114,100,105,110,97,116,111,114,95,116,105,109,101,109,0,0,0,20,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,57,90,108,0,0,0,2,104,3,109,0,0,0,25,55,48,48,53,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,57,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,151,149,213,169,205,231,4,104,3,109,0,0,0,25,54,57,52,52,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,57,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,198,232,168,169,205,231,4,106>>,<<1,131,104,4,109,0,0,0,5,117,115,97,103,101,109,0,0,0,30,111,114,105,103,105,110,97,108,95,100,97,116,97,95,99,111,111,114,100,105,110,97,116,111,114,95,116,105,109,101,109,0,0,0,20,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,48,90,108,0
,0,0,2,104,3,109,0,0,0,25,54,57,56,55,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,48,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,121,143,181,165,205,231,4,104,3,109,0,0,0,25,54,50,48,57,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,57,58,48,48,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,68,222,165,165,205,231,4,106>>,<<1,131,104,4,109,0,0,0,5,117,115,97,103,101,109,0,0,0,30,111,114,105,103,105,110,97,108,95,100,97,116,97,95,99,111,111,114,100,105,110,97,116,111,114,95,116,105,109,101,109,0,0,0,20,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,11,104,3,109,0,0,0,25,55,48,51,48,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,184,197,81,149,205,231,4,104,3,109,0,0,0,25,55,48,48,57,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,161,67,223,145,205,231,4,104,3,109,0,0,0,25,55,48,48,53,59,50,48,49,
51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,225,221,223,145,205,231,4,104,3,109,0,0,0,25,54,57,54,57,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,77,179,222,145,205,231,4,104,3,109,0,0,0,25,54,57,54,55,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,45,134,224,145,205,231,4,104,3,109,0,0,0,25,54,57,53,51,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,11,106,208,145,205,231,4,104,3,109,0,0,0,25,54,57,52,52,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,73,21,225,145,205,231,4,104,3,109,0,0,0,25,54,57,52,50,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,7,0,224,54,222,145,205,231,4,104,3,109,0,0,
0,25,54,57,50,50,59,50,48,49,51,45,49,48,45,48,51,84,48,51,58,49,51,58,50,55,90,108,0,0,0,1,104,2,100,0,1,112,107,0,1,0,106,110,...>>,...],...},...}
2013-12-03 14:35:45.454 [error]
<0.20959.37>@riak_core_handoff_sender:start_fold:269 hinted_handoff transfer of
riak_search_vnode from 'riak@10.196.50.172'
502391187832497878132516661246222288006726811648 to 'riak@10.197.0.55'
502391187832497878132516661246222288006726811648 failed because of
error:{badrecord,ho_acc}
[{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,193}]}]
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
