The process is almost the same as bootstrapping. The leaving node state transits from NORMAL to LEAVING and finally to LEFT. It waits for the ring delay as part of each state transition in order to propagate the entire cluster. Pending ranges are updated. In the case of leaving, there will be nodes responsible for more ranges (in addition to their current ranges). Those nodes and the leaving node keep on receiving writes until LEFT.
- Yifan On Thu, Feb 4, 2021 at 1:39 PM Han <keepsim...@gmail.com> wrote: > Thank you Yifan for the details. I have a related question: when we issue > a command to remove a node A from the ring, there could be a time that > some node B thinks node A is removed, but some node C still thinks node A > is in the ring and could reach node A. What happens if node C sends a > write request to node A during this time? Is it possible to happen? > > Thanks > Han > > > On Wed, Jan 27, 2021 at 5:48 PM Yifan Cai <yc25c...@gmail.com> wrote: > >> Your thoughts regarding Gossip are correct. There could be a time that >> nodes in the cluster hold different views of the ring locally. >> >> In the case of bootstrapping, >> 1. The joining node updates its status to BOOT before streaming data and >> waits for a certain delay in order to populate the update in the cluster. >> 2. The replica nodes calculate the pending ranges, and the joining node >> can get new writes for the token ranges, meanwhile data streaming is >> on-going. >> 3. When the other nodes learn the joining node has finished bootstrapping >> via gossip (after some delay), the new node starts to serve the read as >> well. >> >> (I was wrong about not getting any client traffic earlier. The joining >> node accepts write, but no read) >> >> - Yifan >> >> On Tue, Jan 26, 2021 at 11:36 AM Han <keepsim...@gmail.com> wrote: >> >>> >>>>> I'm particularly trying to understand the fault-tolerant part of >>>>> updating Token Ring state on every node >>>> >>>> The new node only joins the ring (updates the rings state) when the >>>> data streaming (bootstrapping) is successful. Otherwise, the existing ring >>>> remains as is, the joining node remains in JOINING state, and it won't get >>>> any client traffic. If I understand the question correctly. >>>> >>> >>> Thanks Yifan for your response. >>> >>> The ring state update is propagated via gossip, so it is "eventually >>> consistent". This means there is a time period when some existing node has >>> the new token ring but other existing nodes still have the old ring info, >>> right? Is it true that other operations (e.g. replications) are still >>> going on between nodes, even when their local token rings info are not >>> consistent? >>> >>> From the new node point of view, even when it is successful, at the end >>> of `StorageService::joinTokenRing`, it is possible that some existing nodes >>> have not updated their token ring yet, is this correct? >>> >>> Thanks >>> Han >>> >>> >>> >>>> >>>> Hopefully, the answers help. >>>> >>>> - Yifan >>>> >>>> On Sun, Jan 24, 2021 at 1:00 PM Han <keepsim...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I wanted to understand how the bootstrapping (add a new node) works. >>>>> My understanding is that the first step is Token Allocation and the new >>>>> node will get a number of tokens. >>>>> >>>>> My question is: >>>>> >>>>> How / when do the existing nodes update their Token Ring state? and >>>>> is that different between the seed node and non-seed node? >>>>> >>>>> I'm particularly trying to understand the fault-tolerant part of >>>>> updating Token Ring state on every node, but couldn't find relevant info >>>>> by >>>>> searching. >>>>> >>>>> Any info or pointers are appreciated. >>>>> >>>>> Thanks! >>>>> Han >>>>> >>>>>
