Hi Maxim, Basically what Sean suggested is the way to do this without downtime.
To clarify the, the *three* steps following the "Decommission each node in the DC you are working on" step should be applied to *only* the decommissioned nodes. So where it say "*all nodes*" or "*every node*" it applies to only the decommissioned nodes. In addition, the step that says "Wipe data on all the nodes", I would delete all files in the following directories on the decommissioned nodes. - data (usually located in /var/lib/cassandra/data) - commitlogs (usually located in /var/lib/cassandra/commitlogs) - hints (usually located in /var/lib/casandra/hints) - saved_caches (usually located in /var/lib/cassandra/saved_caches) Cheers, Anthony On Fri, 31 Jan 2020 at 03:05, Durity, Sean R <sean_r_dur...@homedepot.com> wrote: > Your procedure won’t work very well. On the first node, if you switched to > 4, you would end up with only a tiny fraction of the data (because the > other nodes would still be at 256). I updated a large cluster (over 150 > nodes – 2 DCs) to smaller number of vnodes. The basic outline was this: > > > > - Stop all repairs > - Make sure the app is running against one DC only > - Change the replication settings on keyspaces to use only 1 DC > (basically cutting off the other DC) > - Decommission each node in the DC you are working on. Because the > replication setting are changed, no streaming occurs. But it releases the > token assignments > - Wipe data on all the nodes > - Update configuration on every node to your new settings, including > auto_bootstrap = false > - Start all nodes. They will choose tokens, but not stream any data > - Update replication factor for all keyspaces to include the new DC > - I disabled binary on those nodes to prevent app connections > - Run nodetool reduild with -dc (other DC) on as many nodes as your > system can safely handle until they are all rebuilt. > - Re-enable binary (and app connections to the rebuilt DC) > - Turn on repairs > - Rest for a bit, then reverse the process for the remaining DCs > > > > > > > > Sean Durity – Staff Systems Engineer, Cassandra > > > > *From:* Maxim Parkachov <lazy.gop...@gmail.com> > *Sent:* Thursday, January 30, 2020 10:05 AM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] How to reduce vnodes without downtime > > > > Hi everyone, > > > > with discussion about reducing default vnodes in version 4.0 I would like > to ask, what would be optimal procedure to perform reduction of vnodes in > existing 3.11.x cluster which was set up with default value 256. Cluster > has 2 DC with 5 nodes each and RF=3. There is one more restriction, I could > not add more servers, nor to create additional DC, everything is physical. > This should be done without downtime. > > > > My idea for such procedure would be > > > > for each node: > > - decommission node > > - set auto_bootstrap to true and vnodes to 4 > > - start and wait till node joins cluster > > - run cleanup on rest of nodes in cluster > > - run repair on whole cluster (not sure if needed after cleanup) > > - set auto_bootstrap to false > > repeat for each node > > > > rolling restart of cluster > > cluster repair > > > > Is this sounds right ? My concern is that after decommission, node will > start on the same IP which could create some confusion. > > > > Regards, > > Maxim. > > ------------------------------ > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. >
