Didn't know that about auto_bootstrap and the algorithm. We should probably fix that. Can you create a JIRA for that issue? Workaround for #2 would be to truncate system.available_ranges after "bootstrap".
On 17 January 2018 at 17:26, Oleksandr Shulgin <oleksandr.shul...@zalando.de > wrote: > On Wed, Jan 17, 2018 at 4:21 AM, kurt greaves <k...@instaclustr.com> > wrote: > >> I believe you are able to get away with just altering the keyspace to >> include both DC's even before the DC exists, and then adding your nodes to >> that new DC using the algorithm. Note you'll probably want to take the >> opportunity to reduce the number of vnodes to something reasonable. Based >> off memory from previous testing you can get a good token balance with 16 >> vnodes if you have at least 6 nodes per rack (with RF=3 and 3 racks). >> > > Alexander, Kurt, > > Thank you for the suggestions. > > None of them did work in the end, unfortunately: > > 1. Using auto_bootstrap=false always results in random token allocation, > ignoring the allocate_tokens_for_keyspace option. > > The token allocation option is only considered if shouldBootstrap() > returns true: > https://github.com/apache/cassandra/blob/cassandra-3.0. > 15/src/java/org/apache/cassandra/service/StorageService.java#L790 if > (shouldBootstrap()) { > https://github.com/apache/cassandra/blob/cassandra-3.0. > 15/src/java/org/apache/cassandra/service/StorageService.java#L842 > BootStrapper.getBootstrapTokens() (the only place in code using the > token allocation option) > https://github.com/apache/cassandra/blob/cassandra-3.0. > 15/src/java/org/apache/cassandra/service/StorageService.java#L901 else { > ... > > 2. Using auto_bootstrap=true and allocate_tokens_for_keyspace=data_ks > gives us balanced range ownership on the new empty DC. The problem though, > is that rebuilding of an already bootstrapped node doesn't work: the node > believes that it already has all the data. > > We are going to proceed by manually assigning a small number of tokens to > the nodes in new DC with auto_bootstrap=false and only use the automatic > token allocation when we need to scale it out. This seems to be the only > supported way to use it anyway. > > Regards, > -- > Alex > >
