Hi All -
I’m having a lot of trouble adding a new DC and getting a balanced ring (i.e.
every node has the same percentage of the token ring).
My config:
GossipingPropertyFileSnitch
allocate_tokens_for_keyspace: <points to a NetworkTopologyStrategy RF=3
keyspace in the existing DC>
num_tokens = 16
6 nodes in the new DC / 3 nodes in the existing DC
Cassandra 3.0.23
I add the nodes to the new DC one-by-one, waiting for “Startup complete” … then
create a new test keyspace with RF=3:
create keyspace test_tokens with replication = {'class':
'NetworkTopologyStrategy', 'ies3': '3'}
… but then when I run “nodetool status test_tokens”, i see that the “Owns
(effective)” is way out of balance (see attached image — “ies3” is the new DC).
*.62 / node1 / rack1 - 71.8%
*.63 / node2 / rack2 - 91.4%
*.64 / node3 / rack3 - 91.6%
*.66 / node4 / rack1 - 28.2%
*.67 / node5 / rack2 - 8.6%
*.68 / node6 / rack3 - 8.4%
node1 & node2 are seed nodes, along with 2 nodes from the existing DC.
How can I get even token distribution — “Owns (effective) = 50%" (or 1/6 of the
token range for each node)?
Also: I’ve made several attempts to try to figure this out (ex: all nodes in 1
rack? each node has own rack? 2 nodes per rack?). Between each attempt I’m
running “nodetool decommission” one-by-one, blowing away /var/lib/cassandra/*,
etc. Is it possible that the existing DC’s gossip is remembering the token
range & thus causing problems when I recreate the new DC with some other
configuration parameters? Do I need to do something to clear out the gossip
between attempts?
Thanks everyone.
- Max
