Oleksandr,
Could it be that after distributing the data, some of the nodes did not need to have a fourth token? Kenneth Brotman From: Oleksandr Shulgin [mailto:oleksandr.shul...@zalando.de] Sent: Thursday, January 25, 2018 3:44 AM To: User Subject: Nodes show different number of tokens than initially Hello, While testing token allocation with version 3.0.15 we are experiencing some quite unexpected result. We have deployed a secondary virtual DC with 6 nodes, 4 tokens per node. Then we were adding the 7th node to the new DC in order to observe the effect of ownership re-distribution. To set up the new DC we've used the following steps: 1. Alter all keyspaces to replicate to the upcoming new DC. 2. Deploy 3 seed nodes (IP ends with .31) with num_tokens=4 and tokens specified by initial_token list, auto_bootstrap=false. 3. Deploy 3 more nodes (IP ends with .32) with num_tokens=4 and allocate_tokens_for_keyspace=data_ks, auto_bootstrap=true. 4. Rebuild all new nodes specifying eu-central as the source DC (for the 3 already bootstrapped nodes, workaround by truncating system.available_ranges first). The following is the output of nodetool status after starting to bootstrap the 7th node (172.31.128.33): Datacenter: eu-central ====================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.31.160.12 26.4 GB 256 48.9% 89067222-b0eb-49e5-be7d-758ea24ace9a 1c UN 172.31.144.12 28.92 GB 256 52.6% 2ab4786f-9722-4418-ba78-9c435cbb30e5 1b UN 172.31.128.12 28.13 GB 256 47.9% c4733a5c-abc5-4bab-9449-1e3f584cf64f 1a UN 172.31.128.11 29.84 GB 256 52.2% 6083369c-1a0f-4098-a420-313dacd429b6 1a UN 172.31.160.11 28.25 GB 256 51.1% 4dc361fc-818a-4b7f-abd3-9121488a7db1 1c UN 172.31.144.11 28.14 GB 256 47.4% 05e5df92-d196-46d5-8812-e843fbbd2922 1b Datacenter: eu-central_4vn ========================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.31.128.31 24.83 GB 4 45.8% 4d7decb3-8692-4aec-a2e1-2ac89aed8c5a 1a UN 172.31.144.31 26.52 GB 4 45.8% 2eb29602-2df5-4f4f-b419-b5a94cf785f0 1b UN 172.31.160.31 248 GB 4 45.8% f1bd4696-c25c-4bc3-8c30-292f2bd027c1 1c UJ 172.31.128.33 568.94 MB 4 ? ffa21d50-9bb4-4d2b-9e3e-7a6945f6f071 1a UN 172.31.144.32 29.3 GB 4 54.2% 5ce019f6-99fd-4333-b231-d04a266229bb 1b UN 172.31.160.32 27.8 GB 4 54.2% 193bef27-eea8-4aa6-9d5f-8baf3decdd76 1c UN 172.31128.32 30.5 GB 4 54.2% 6a046b64-31f9-4881-85b0-ab3a2f6dcdc4 1a Then we wanted to start testing distribution with 8 vnodes. For that we started to deploy yet another DC. The following is the output of nodetool status after deploying the 3 seed nodes of the 8-tokens DC: Datacenter: eu-central ====================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.31.160.12 26.4 GB 256 48.9% 89067222-b0eb-49e5-be7d-758ea24ace9a 1c UN 172.31.144.12 28.92 GB 256 52.6% 2ab4786f-9722-4418-ba78-9c435cbb30e5 1b UN 172.31.128.12 28.13 GB 256 47.9% c4733a5c-abc5-4bab-9449-1e3f584cf64f 1a UN 172.31.128.11 29.84 GB 256 52.2% 6083369c-1a0f-4098-a420-313dacd429b6 1a UN 172.31.160.11 28.25 GB 256 51.1% 4dc361fc-818a-4b7f-abd3-9121488a7db1 1c UN 172.31.144.11 28.14 GB 256 47.4% 05e5df92-d196-46d5-8812-e843fbbd2922 1b Datacenter: eu-central_4vn ========================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.31.128.31 24.83 GB 3 45.8% 4d7decb3-8692-4aec-a2e1-2ac89aed8c5a 1a UN 172.31.144.31 26.52 GB 4 45.8% 2eb29602-2df5-4f4f-b419-b5a94cf785f0 1b UN 172.31.160.31 24.8 GB 4 45.8% f1bd4696-c25c-4bc3-8c30-292f2bd027c1 1c UJ 172.31.128.33 4.21 GB 4 ? ffa21d50-9bb4-4d2b-9e3e-7a6945f6f071 1a UN 17231.160.32 27.8 GB 4 54.2% 193bef27-eea8-4aa6-9d5f-8baf3decdd76 1c UN 172.31.144.32 29.3 GB 3 54.2% 5ce019f6-99fd-4333-b231-d04a266229bb 1b UN 172.31.128.32 30.5 GB 4 54.2% 6a046b64-31f9-4881-85b0-ab3a2f6dcdc4 1a Datacenter: eu-central_8vn ========================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 172.31.128.41 111.11 KB 8 0.0% e218b68e-9837-4e6a-acbe-9833fda285bc 1a UN 172.31.144.41 113.2 KB 8 0.0% 3ec883e9-6b84-4314-85bd-a3c00c4f47c8 1b UN 172.31.160.41 82.22 KB 8 0.0% cfaee6c5-ee9c-4d29-aa54-ca3e8e74e356 1c What is absolutely unexpected is that here we see that 2 nodes in the _4vn DC apparently now have reduced number of tokens: 3 instead of 4. How could that happen? -- Alex
