Thank you Yifan for the details. I have a related question: when we issue a command to remove a node A from the ring, there could be a time that some node B thinks node A is removed, but some node C still thinks node A is in the ring and could reach node A. What happens if node C sends a write request to node A during this time? Is it possible to happen?
Thanks Han On Wed, Jan 27, 2021 at 5:48 PM Yifan Cai <yc25c...@gmail.com> wrote: > Your thoughts regarding Gossip are correct. There could be a time that > nodes in the cluster hold different views of the ring locally. > > In the case of bootstrapping, > 1. The joining node updates its status to BOOT before streaming data and > waits for a certain delay in order to populate the update in the cluster. > 2. The replica nodes calculate the pending ranges, and the joining node > can get new writes for the token ranges, meanwhile data streaming is > on-going. > 3. When the other nodes learn the joining node has finished bootstrapping > via gossip (after some delay), the new node starts to serve the read as > well. > > (I was wrong about not getting any client traffic earlier. The joining > node accepts write, but no read) > > - Yifan > > On Tue, Jan 26, 2021 at 11:36 AM Han <keepsim...@gmail.com> wrote: > >> >>>> I'm particularly trying to understand the fault-tolerant part of >>>> updating Token Ring state on every node >>> >>> The new node only joins the ring (updates the rings state) when the data >>> streaming (bootstrapping) is successful. Otherwise, the existing ring >>> remains as is, the joining node remains in JOINING state, and it won't get >>> any client traffic. If I understand the question correctly. >>> >> >> Thanks Yifan for your response. >> >> The ring state update is propagated via gossip, so it is "eventually >> consistent". This means there is a time period when some existing node has >> the new token ring but other existing nodes still have the old ring info, >> right? Is it true that other operations (e.g. replications) are still >> going on between nodes, even when their local token rings info are not >> consistent? >> >> From the new node point of view, even when it is successful, at the end >> of `StorageService::joinTokenRing`, it is possible that some existing nodes >> have not updated their token ring yet, is this correct? >> >> Thanks >> Han >> >> >> >>> >>> Hopefully, the answers help. >>> >>> - Yifan >>> >>> On Sun, Jan 24, 2021 at 1:00 PM Han <keepsim...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I wanted to understand how the bootstrapping (add a new node) works. My >>>> understanding is that the first step is Token Allocation and the new node >>>> will get a number of tokens. >>>> >>>> My question is: >>>> >>>> How / when do the existing nodes update their Token Ring state? and is >>>> that different between the seed node and non-seed node? >>>> >>>> I'm particularly trying to understand the fault-tolerant part of >>>> updating Token Ring state on every node, but couldn't find relevant info by >>>> searching. >>>> >>>> Any info or pointers are appreciated. >>>> >>>> Thanks! >>>> Han >>>> >>>>
