On Wed, Feb 13, 2019 at 7:47 AM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote:
> On Wed, Feb 13, 2019 at 4:40 PM Jeff Jirsa <jji...@gmail.com> wrote: > >> Some people who add new hosts rebalance the ring afterward - that >> rebalancing can look a lot like a shrink. >> > > You mean by moving the tokens? That's only possible if one is not using > vnodes, correct? > Believe that's correct (I'm not sure how to move individual tokens in a vnode cluster, at least, short of adding a host with exact token placement you want a decommissioning another host). > > I also believe, but don’t have time to prove, that enough new hosts can >> eventually give you a range back (moving it all the way around the ring) - >> less likely but probably possible. >> >> Easiest to just assume that any range movement may resurrect data if you >> haven’t run cleanup. >> > > Does this mean that it is recommended to run cleanup on all hosts after > every single node added? We currently do this after every 3 or 6 nodes (1 > or 2 new per rack), to minimize the number of times we have to rewrite the > sstable files. Arguably, we don't do explicit deletes, the data is only > expiring due to TTL, so this should not be a problem for us, but in general? > Depending on how bad data resurrection is, you should run it for any host that loses a range. In vnodes, that's usually all hosts. Cleanup with LCS is very cheap. Cleanup with STCS/TWCS is a bit more work. If you're just TTL'ing all data, it may not be worth the effort.
