I'm wondering how the Cassandra protocol brings a newly bootstrapped node "up to speed".
for ease of illustration, let's say we just have one key, K, and the value is continually updated: 1,2 ,3 ,4 .... originally we have 1 node, A, now node B joins, and needs to bootstrap and get its newly assigned range (just "K") from A. now let's say A has seen updates 1,2,3 up to this point. according to the StreamingRequestVerbHandler , A does a flush of its memtable, then streams out the new sstables. but what while the (newly-flushed) sstable is being streamed out from A, before B fully received them, A now gets more updates: 4,5,6.... ? now B gets the streamed range, and happily declares itself ready, and joins the ring. but now it's actually not "up to speed" with the "old members". cuz A now has a value K=6 while B has K=3 of course when clients query now, A's and B's results are reconciled, so client gets latest result. but would B stay forever "not up to speed" ? how can we make it up to speed? cuz although the following is a very hypothetical scenario, it will lead to lost writes: say B is still in the "not up to date " state, then another node is removed and a new node is inserted, then after more of such cycles, all the "up to date" nodes are gone, and we essentially lose the latest writes.
