Important: Removing/leaving a node from the cluster does NOT automatically
replicate the data that was on that node to get back up to your desired
replication level.
Right now, the suggested method for doing that involves invoking a read on all
of the keys that were on that node. This forces a read-repair which will cause
the data to replicate out to nodes that are still alive and in the cluster.
Something like this:
Do a list-keys on the node you intend to remove, save this file.
Remove the node to be decommissioned.
Add a new node.
Issue a read on every key from the list you created.
However, the list-keys method in 1.0 is not complete if you have lost a node.
It requires all nodes to be up and running. Effectively, there is no way right
now to guarantee your replication level on your data in case of a node
failure/unexpected removal.
This is supposed to be addressed in 1.1:
https://issues.basho.com/show_bug.cgi?id=991
Also, fwiw, this information came out of a ticket Bump had with Basho to try to
understand this better. I may be misquoting, and if so, I hope Sean or someone
will step in and correct me. :)
--
Mark Smith // m...@bu.mp
Operations Lead
On Monday, October 24, 2011 at 10:06 AM, Tomer Naor wrote:
>
> Well, it caused us some problems…
>
>
>
>
>
> The situation is as follows:
>
>
> we have a production cluster with five 0.14 nodes and which we want to
> replace with three new Riak 1.0.1 servers.
>
>
>
>
>
> This is what we did:
>
>
> 1. join each of the 1.0 nodes to the 0.14 cluster.
>
>
> 2. after that all the three 1.0 nodes were part of the ring members and
> handoff was over we did ‘admin-riak leave’ on one of the 0.14 nodes
> (eventually we’ll need to leave them all).
>
>
>
>
>
> The problem is that it looks like not all the data from the 0.14 node were
> handoff in the leaving process and it seems like we’ve lost some data.
>
>
> (Don’t know if it’s matter but the bitcask of the 0.14 node that we tried to
> ‘riak-admin leave’ was reduced from 120GB to 55GB)
>
>
>
>
>
> Another problem that we noticed is that the basic get api not always returned
> a consistent response, it sometimes returned the required data and sometimes
> returned 404 status code.
>
>
>
>
>
> what is the best/right/safe way to do it without losing data and without
> experience significant downtime as a result of the join and leave process.
>
>
>
>
>
> Thanks,
>
>
> Tomer.
>
>
>
>
>
> From: Sean Cribbs [mailto:s...@basho.com]
> Sent: Wednesday, October 12, 2011 14:37
> To: Tomer Naor
> Cc: riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> Subject: Re: Join a Riak-1.0 node to a Riak-0.14 cluster
>
>
>
>
>
>
> It is possible but requires a slight modification from the directions in
> http://wiki.basho.com/Rolling-Upgrades.html. When adding the new 1.0 node,
> make sure these settings are in the 'riak_kv' section of its app.config:
>
>
>
>
>
>
> {legacy_keylisting, true},
> {mapred_system, legacy},
> {vnode_vclocks, false}
>
> This will ensure that it does not try to use functionality that is
> unavailable on the 0.14.2 nodes.
>
>
>
>
>
>
> On Wed, Oct 12, 2011 at 6:30 AM, Tomer Naor <to...@conduit.com
> (mailto:to...@conduit.com)> wrote:
>
>
> Hi,
>
>
>
>
>
> Is it possible to join a Riak-1.0 node (not from rolling upgrade - new server
> with fresh 1.0.0 installation) to a cluster with Riak-0.14 nodes without any
> unexpected problems?
>
>
>
>
>
> Thanks,
>
>
> Tomer.
>
>
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Sean Cribbs <s...@basho.com (mailto:s...@basho.com)>
>
>
> Developer Advocate
>
>
>
> Basho Technologies, Inc.
>
>
>
> http://www.basho.com/
>
>
>
>
>
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
