Hello dear list.
Recently we had an issue: 1 from 4 of our riak nodes went down
('[email protected]'). As expected 3 other nodes took it's work and everything
was ok.
After some time we was able to recover this node and put it again into
cluster.
But now I see some strange things in logs:
# riak-admin transfers
Attempting to restart script through sudo -H -u riak
'[email protected]' waiting to handoff 1 partitions
>From '[email protected]' logs:
[error] hinted_handoff transfer of riak_kv_vnode from '[email protected]'
1244559988039597016282825365359959758925755056128 to '[email protected]'
1244559988039597016282825365359959758925755056128 failed because of
error:{case_clause,{error,closed}}
[{riak_core_handoff_sender,start_fold,5,[{file,"src/riak_core_handoff_sender.erl"},{line,174}]}]
And from '[email protected]' (recovered node) logs:
2013-08-01 08:20:00.222 UTC [error] <0.31615.12> Supervisor poolboy_sup had
child riak_core_vnode_worker started with
riak_core_vnode_worker:start_link([{worker_args,[1244559988039597016282825365359959758925755056128,[],worker_props]},{worker_callback_mod,...},...])
at undefined exit with reason no case clause matching
{error,bad_crc,{state,#Ref<0.0.11.259591>,"1244559988039597016282825365359959758925755056128",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{reduce_js_vm_count,0},{js_thread_stack,16},{riak_kv_stat,true},{map_js_vm_count,0},{mapred_system,pipe},{js_max_vm_mem,8},{legacy_stats,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{http_url_encoding,...},...],...}}
in riak_kv_vnode:prepare_put/3 line 698 in context shutdown_error
2013-08-01 08:20:00.231 UTC [error] <0.31615.12> gen_server <0.31615.12>
terminated with reason: no case clause matching
{error,bad_crc,{state,#Ref<0.0.11.259591>,"1244559988039597016282825365359959758925755056128",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{reduce_js_vm_count,0},{js_thread_stack,16},{riak_kv_stat,true},{map_js_vm_count,0},{mapred_system,pipe},{js_max_vm_mem,8},{legacy_stats,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{http_url_encoding,...},...],...}}
in riak_kv_vnode:prepare_put/3 line 698
2013-08-01 08:20:00.255 UTC [error] <0.31615.12> CRASH REPORT Process
<0.31615.12> with 0 neighbours exited with reason: no case clause matching
{error,bad_crc,{state,#Ref<0.0.11.259591>,"1244559988039597016282825365359959758925755056128",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{reduce_js_vm_count,0},{js_thread_stack,16},{riak_kv_stat,true},{map_js_vm_count,0},{mapred_system,pipe},{js_max_vm_mem,8},{legacy_stats,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{http_url_encoding,...},...],...}}
in riak_kv_vnode:prepare_put/3 line 698 in gen_server:terminate/6 line 747
It seems that for some reason '[email protected]' node abort handoff every
time '[email protected]' tried to give it's data back.
Riak version is 1.2.1-1
--
View this message in context:
http://riak-users.197444.n3.nabble.com/riak-hinted-handoff-freezed-transfer-tp4028652.html
Sent from the Riak Users mailing list archive at Nabble.com.
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
