We have a 1.0.2 cluster with a node that's gone but still listed in member_status (as legacy):
riak-admin member_status Attempting to restart script through sudo -u riak ================================= Membership ================================== Status Ring Pending Node ------------------------------------------------------------------------------- (legacy) 7.8% -- 'riak@10.115.13.51' valid 7.8% -- 'riak@10.119.82.164' valid 7.8% -- 'riak@10.13.22.183' valid 7.8% -- 'riak@10.13.51.171' valid 7.8% -- 'riak@10.37.46.7' valid 7.6% -- 'riak@10.76.45.111' valid 7.6% -- 'riak@10.76.62.122' valid 7.6% -- 'riak@10.78.197.82' valid 7.6% -- 'riak@10.79.69.234' valid 7.6% -- 'riak@10.80.157.112' valid 7.6% -- 'riak@10.82.155.202' valid 7.6% -- 'riak@10.82.25.84' valid 7.6% -- 'riak@10.84.5.68' ------------------------------------------------------------------------------- Valid:13 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 riak-admin force-remove does not remove the node (we've tried a few times over the last 4 days). This node was down before we did the upgrade, but wasn't removing so we upgraded anyway. As you can see here, doing a force-remove reports successful, but the node is still listed: riak-admin force-remove 'riak@10.115.13.51' Attempting to restart script through sudo -u riak Success: "riak@10.115.13.51" removed from the cluster riak-admin member_status Attempting to restart script through sudo -u riak ================================= Membership ================================== Status Ring Pending Node ------------------------------------------------------------------------------- (legacy) 7.8% -- 'riak@10.115.13.51' valid 7.8% -- 'riak@10.119.82.164' valid 7.8% -- 'riak@10.13.22.183' valid 7.8% -- 'riak@10.13.51.171' valid 7.8% -- 'riak@10.37.46.7' valid 7.6% -- 'riak@10.76.45.111' valid 7.6% -- 'riak@10.76.62.122' valid 7.6% -- 'riak@10.78.197.82' valid 7.6% -- 'riak@10.79.69.234' valid 7.6% -- 'riak@10.80.157.112' valid 7.6% -- 'riak@10.82.155.202' valid 7.6% -- 'riak@10.82.25.84' valid 7.6% -- 'riak@10.84.5.68' ------------------------------------------------------------------------------- Valid:13 / Leaving:0 / Exiting:0 / Joining:0 / Down:0 After this, a small number of handoffs are seen in the logs: 17:13:29.514 [info] Handoff of partition riak_kv_vnode 1033327329519114953886199041881434478741108555776 from 'riak@10.13.51.171' to 'riak@10.82.25.84' completed: sent 2 objects in 0.04 seconds 17:13:39.470 [info] Starting handoff of partition riak_kv_vnode 176978713895539025251227460211737396911460581376 from 'riak@10.13.51.171' to 'riak@10.82.155.202' 17:13:39.480 [info] Starting handoff of partition riak_kv_vnode 1238850997268773176758592221482161778380224069632 from 'riak@10.13.51.171' to 'riak@10.76.45.111' 17:13:39.512 [info] Handoff of partition riak_kv_vnode 176978713895539025251227460211737396911460581376 from 'riak@10.13.51.171' to 'riak@10.82.155.202' completed: sent 1 objects in 0.04 seconds 17:13:39.525 [info] Handoff of partition riak_kv_vnode 1238850997268773176758592221482161778380224069632 from 'riak@10.13.51.171' to 'riak@10.76.45.111' completed: sent 3 objects in 0.04 seconds Here's the pending transfer list: riak-admin transfers Nodes ['riak@10.115.13.51'] are currently down. 'riak@10.84.5.68' waiting to handoff 28 partitions 'riak@10.82.25.84' waiting to handoff 40 partitions 'riak@10.82.155.202' waiting to handoff 40 partitions 'riak@10.80.157.112' waiting to handoff 40 partitions 'riak@10.79.69.234' waiting to handoff 40 partitions 'riak@10.78.197.82' waiting to handoff 39 partitions 'riak@10.76.62.122' waiting to handoff 40 partitions 'riak@10.76.45.111' waiting to handoff 40 partitions 'riak@10.37.46.7' waiting to handoff 40 partitions 'riak@10.13.51.171' waiting to handoff 40 partitions 'riak@10.13.22.183' waiting to handoff 40 partitions 'riak@10.119.82.164' waiting to handoff 40 partitions Any ideas on how to get the cluster to remove this node? Thanks, Paul _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
