Hi Enrico, Glad to hear the problem has been resolved and thank you for the feedback!
Kind regards, Anthony On Mon, 2 Dec 2019 at 22:03, Enrico Cavallin <cavallin.enr...@gmail.com> wrote: > Hi Anthony, > thank you for your hints, now the new DC is well balanced within 2%. > I did read your article, but I thought it was needed only for new > "clusters", not also for new "DCs"; but RF is per DC so it makes sense. > > You TLP guys are doing a great job for Cassandra community. > > Thank you, > Enrico > > > On Fri, 29 Nov 2019 at 05:09, Anthony Grasso <anthony.gra...@gmail.com> > wrote: > >> Hi Enrico, >> >> This is a classic chicken and egg problem with the >> allocate_tokens_for_keyspace setting. >> >> The allocate_tokens_for_keyspace setting uses the replication factor of >> a DC keyspace to calculate the token allocation when a node is added to the >> cluster for the first time. >> >> Nodes need to be added to the new DC before we can replicate the keyspace >> over to it. Herein lies the problem. We are unable to use >> allocate_tokens_for_keyspace unless the keyspace is replicated to the >> new DC. In addition, as soon as you change the keyspace replication to the >> new DC, new data will start to be written to it. To work around this issue >> you will need to do the following. >> >> 1. Decommission all the nodes in the *dcNew*, one at a time. >> 2. Once all the *dcNew* nodes are decommissioned, wipe the contents >> in the *commitlog*, *data*, *saved_caches*, and *hints* directories >> of these nodes. >> 3. Make the first node to add into the *dcNew* a seed node. Set the >> seed list of the first node with its IP address and the IP addresses of >> the >> other seed nodes in the cluster. >> 4. Set the *initial_token* setting for the first node. You can >> calculate the values using the algorithm in my blog post: >> >> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html. >> For convenience I have calculated them: >> *-9223372036854775808,-4611686018427387904,0,4611686018427387904*. >> Note, remove the *allocate_tokens_for_keyspace* setting from the >> *cassandra.yaml* file for this (seed) node. >> 5. Check to make sure that no other node in the cluster is assigned >> any of the four tokens specified above. If there is another node in the >> cluster that is assigned one of the above tokens, increment the >> conflicting >> token by values of one until no other node in the cluster is assigned that >> token value. The idea is to make sure that these four tokens are unique to >> the node. >> 6. Add the seed node to cluster. Make sure it is listed in *dcNew *by >> checking nodetool status. >> 7. Create a dummy keyspace in *dcNew* that has a replication factor >> of 2. >> 8. Set the *allocate_tokens_for_keyspace* value to be the name of the >> dummy keyspace for the other two nodes you want to add to *dcNew*. >> Note remove the *initial_token* setting for these other nodes. >> 9. Set *auto_bootstrap* to *false* for the other two nodes you want >> to add to *dcNew*. >> 10. Add the other two nodes to the cluster, one at a time. >> 11. If you are happy with the distribution, copy the data to *dcNew* >> by running a rebuild. >> >> >> Hope this helps. >> >> Regards, >> Anthony >> >> On Fri, 29 Nov 2019 at 02:08, Enrico Cavallin <cavallin.enr...@gmail.com> >> wrote: >> >>> Hi all, >>> I have an old datacenter with 4 nodes and 256 tokens each. >>> I am now starting a new datacenter with 3 nodes and num_token=4 >>> and allocate_tokens_for_keyspace=myBiggestKeyspace in each node. >>> Both DCs run Cassandra 3.11.x. >>> >>> myBiggestKeyspace has RF=3 in dcOld and RF=2 in dcNew. Now dcNew is very >>> unbalanced. >>> Also keyspaces with RF=2 in both DCs have the same problem. >>> Did I miss something or even with allocate_tokens_for_keyspace I have >>> strong limitations with low num_token? >>> Any suggestions on how to mitigate it? >>> >>> # nodetool status myBiggestKeyspace >>> Datacenter: dcOld >>> ======================= >>> Status=Up/Down >>> |/ State=Normal/Leaving/Joining/Moving >>> -- Address Load Tokens Owns (effective) Host ID >>> Rack >>> UN x.x.x.x 515.83 GiB 256 76.2% >>> fc462eb2-752f-4d26-aae3-84cb9c977b8a rack1 >>> UN x.x.x.x 504.09 GiB 256 72.7% >>> d7af8685-ba95-4854-a220-bc52dc242e9c rack1 >>> UN x.x.x.x 507.50 GiB 256 74.6% >>> b3a4d3d1-e87d-468b-a7d9-3c104e219536 rack1 >>> UN x.x.x.x 490.81 GiB 256 76.5% >>> 41e80c5b-e4e3-46f6-a16f-c784c0132dbc rack1 >>> >>> Datacenter: dcNew >>> ============== >>> Status=Up/Down >>> |/ State=Normal/Leaving/Joining/Moving >>> -- Address Load Tokens Owns (effective) Host ID >>> Rack >>> UN x.x.x.x 145.47 KiB 4 56.3% >>> 7d089351-077f-4c36-a2f5-007682f9c215 rack1 >>> UN x.x.x.x 122.51 KiB 4 55.5% >>> 625dafcb-0822-4c8b-8551-5350c528907a rack1 >>> UN x.x.x.x 127.53 KiB 4 88.2% >>> c64c0ce4-2f85-4323-b0ba-71d70b8e6fbf rack1 >>> >>> Thanks, >>> -- ec >>> >>
