On Tue, Jan 30, 2018 at 5:13 AM, kurt greaves <k...@instaclustr.com> wrote:
> Shouldn't happen. Can you send through nodetool ring output from one of > those nodes? Also, did the logs have anything to say about tokens when you > started the 3 seed nodes? > Hi Kurt, I cannot run nodetool ring anymore, since these test nodes are long gone. However I've grepped the logs and this is what I've found: Jan 25 08:57:18 ip-172-31-128-41 docker/cf3ea463915a[854]: INFO 08:57:18 Nodes /172.31.128.31 and /172.31.128.41 have the same token -9223372036854775808. Ignoring /172.31.128.31 Jan 25 08:57:18 ip-172-31-128-41 docker/cf3ea463915a[854]: INFO 08:57:18 Nodes /172.31.144.32 and /172.31.128.41 have the same token -8454757700450211158. Ignoring /172.31.144.32 Jan 25 08:58:30 ip-172-31-144-41 docker/48fba443d99f[852]: INFO 08:58:30 Nodes /172.31.128.41 and /172.31.128.31 have the same token -9223372036854775808. /172.31.128.41 is the new owner Jan 25 08:58:30 ip-172-31-144-41 docker/48fba443d99f[852]: INFO 08:58:30 Nodes /172.31.144.32 and /172.31.128.41 have the same token -8454757700450211158. Ignoring /172.31.144.32 Jan 25 08:59:45 ip-172-31-160-41 docker/cced70e132f2[849]: INFO 08:59:45 Nodes /172.31.128.41 and /172.31.128.31 have the same token -9223372036854775808. /172.31.128.41 is the new owner Jan 25 08:59:45 ip-172-31-160-41 docker/cced70e132f2[849]: INFO 08:59:45 Nodes /172.31.144.32 and /172.31.128.41 have the same token -8454757700450211158. Ignoring /172.31.144.32 Since we are allocating the tokens for seed nodes manually, it appears that the first seed node in the new ring (172.31.128.41) gets the same first token (-9223372036854775808) as the node in the old ring (172.31.128.31). The same goes for the 3rd token of the new seed node (-8454757700450211158). What is beyond me is why would that matter and why would token ownership change at all, while these nodes are in the *different virtual DCs*? To me this sounds like a paticularly nasty bug... -- Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 127-59-707
