Since about a week we run a 16 node Riak cluster. We use Riak 1.1.2 with
Bitcask 1.5.1 as storage backend. We run this on Debian 6.0.5 Linux
2.6.38-bpo.2-amd64 and Erland R14B04.
When we started using this setup we hit the jackpot right away. The disk in one
of the nodes started causing problems quite soon. So we told that machine to
leave the riak cluster.
Unfortunately the handoff had still not completed 12 hours later. In the logs
we saw:
2012-07-13 00:05:34.526 [info]
<0.29522.7>@riak_core_handoff_sender:start_fold:83 Starting handoff of
partition riak_kv_vnode 348248437020254210978221776545676813277390176256 from
'riak@10.53.1.20' to 'riak@10.53.1.15'
2012-07-13 00:06:11.272 [error]
<0.29522.7>@riak_core_handoff_sender:start_fold:119 Handoff of partition
riak_kv_vnode 348248437020254210978221776545676813277390176256 from
'riak@10.53.1.20' to 'riak@10.53.1.15' failed because <0.29523.7> died with
reason
{{badmatch,{error,{bad_crc,<<>>,435787301}}},[{riak_core_handoff_sender,'-start_fold/5-fun-1-',9}]}
2012-07-13 00:06:11.272 [error] emulator Error in process <0.29523.7> on node
'riak@10.53.1.20' with exit value: {{badmatch,{error,{bad_crc,<<0
bytes>>,435787301}}},[{riak_core_handoff_sender,'-start_fold/5-fun-1-',9}]}
Over and over and over again. This is understandable since the disk had a bad
sector. But once stuck in this situation it's quite hard to get Riak unstuck..
We tried to stop the broken node, it wouldn't want to stop nicely so we killed
it with brute force.
Then we tried to run "riak-admin force-remove" from one of the other nodes to
let everybody else know the broken node wouldn't be coming back anytime soon.
Unfortunately it just told us:
Failed: "riak10" is not a member of the cluster.
And the cluster status showed that the broken node was still in state leaving.
Eventually the only way we got it to leave completely was by wiping all the
data on the broken node, letting it rejoin and then issue a force-remove, which
was only possible after it had received all it's data again.
I would say that the force-remove should remove a node from the cluster
regardless the state it's in and it would have far less impact on the other
nodes than to go through the whole rejoin process first.
Anyhow we're back in business. But I thought maybe this can be improved so I
thought it would be good to let you know.
Kind regards,
Erik Hoogeveen
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
