Hi David, There is an excellent article which describes exactly what you want to do (ie migrate from one DC to another DC) : http://planetcassandra.org/blog/cassandra-migration-to-ec2/
2015-03-18 17:05 GMT+01:00 David CHARBONNIER <david.charbonn...@rgsystem.com >: > Hi, > > > > We’re using Cassandra through the Datastax Enterprise package in version > 4.5.1 (Cassandra version 2.0.8.39) with 7 nodes in a single datacenter. > > > > We need to move our Cassandra cluster from France to another country. To > do this, we want to add a second 7-nodes datacenter to our cluster and > stream all data between the two countries before dropping the first > datacenter. > > > > On January 31st, we tried doing so but we had some problems: > > - New nodes in the other country have been installed like French > nodes except for Datastax Enterprise version (4.5.1 in France and 4.6.1 in > the other country which means Cassandra version 2.0.8.39 in France and > 2.0.12.200 in the other country) > > - The following procedure has been followed: > http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html > but an error occurred during step 3. New nodes have been started before > the *cassandra-topology.properties* file has been updated on the original > datacenter. New nodes appeared in the original datacenter instead of the > new one. > > - To recover our original cluster, we decommissionned every node > of the new datacenter with the *nodetool decommission* command. > > > > On February 9th, nodes in the second datacenter have been restarted and > joined the cluster. We had to decommission them just like before. > > > > On February 11th, we added disk space on our 7 running French nodes. To > achieve this, we restarted the cluster but the nodes updated their perring > informations and nodes from Luxembourg (decommissionned on February 9th) > were present. This behaviour is described here: > https://issues.apache.org/jira/browse/CASSANDRA-7825. So we cleaned > *system.peers* table content. > > > > On March 11th, we needed to add an 8th node to our existing French > cluster. We installed the same Datastax Enterprise version (4.5.1 with > Cassandra 2.0.8.39) and tried to add this node to the cluster with this > procedure: > http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html. > In OPSCenter, the node was joining the cluster and data streaming got stuck > at 100%. After several hours, *nodetool status* showed us that the node > was still joining but nothing in the logs let us know there was a problem. We > restarted the node but it has no effect. Then we cleaned data and commitlog > contents and try to add the node to the cluster again but without result. > > Last try was to add the node with *auto_bootstrap : false* in order to > add the node to the cluster manually but it messed up with the data. So we > shut down the node and decommissioned it (with *nodetool removenode*). > The whole cluster has been repaired and we stopped doing anything. > > > > Now, our cluster has only 7 French nodes in which we can’t add any node. The > OPSCenter data has disapeared and we work without any information about how > our cluster is running. > > > > You’ll find attached to this email our current configuration and a > screenshot of our OPSCenter metric page. > > > > Do you have some idea on how to clean up the mess and get our cluster > running cleanly before we start our migration (France to another country > like described in the beginning of this email)? > > > > Thank you. > > > > Best regards, > > > > *David CHARBONNIER* > > Sysadmin > > T : +33 411 934 200 > > david.charbonn...@rgsystem.com > > ZAC Aéroport > > 125 Impasse Adam Smith > > 34470 Pérols - France > > *www.rgsystem.com* <http://www.rgsystem.com/> > > > > > > > -- Fabien Rousseau <aur...@yakaz.com>www.yakaz.com
