Hi Aaron, Thanks for your answer.
I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes instead of 16 (it is a test cluster). There are 2 datacenters b1 and s1. Here is the result of nodetool status after adding a new node in the 1st datacenter (dc s1): root@node007:~# nodetool status Datacenter: b1 ============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.234.72.135 10.71 GB 256 44.6% 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01 UN 10.234.72.134 16.74 GB 256 63.7% f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01 UN 10.234.72.139 17.09 GB 256 62.0% 95661392-ccd8-4592-a76f-1c99f7cdf23a e07 UN 10.234.72.138 10.96 GB 256 42.9% 0d6725f0-1357-423d-85c1-153fb94257d5 e03 UN 10.234.72.137 11.09 GB 256 45.7% 492190d7-3055-4167-8699-9c6560e28164 e03 UN 10.234.72.136 11.91 GB 256 41.1% 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01 Datacenter: s1 ============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.98.255.139 16.94 GB 256 43.8% 3523e80c-8468-4502-b334-79eabc3357f0 g10 UN 10.98.255.138 12.62 GB 256 42.4% a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02 UN 10.98.255.137 10.59 GB 256 38.4% f851b6ee-f1e4-431b-8beb-e7b173a77342 i02 UN 10.98.255.136 11.89 GB 256 42.9% 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09 UN 10.98.255.135 10.29 GB 256 40.4% e2d020a5-97a9-48d4-870c-d10b59858763 a09 UN 10.98.255.134 16.19 GB 256 52.3% 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06 UN 10.98.255.140 127.84 KB 256 39.9% 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10 We can see that the new node (10.98.255.140) contains only 127,84KB. We saw also that there was no network traffic between the nodes. Then we added a new node in the 2nd datacenter (dc b1) root@node007:~# nodetool status Datacenter: b1 ============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.234.72.135 12.95 GB 256 42.0% 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01 UN 10.234.72.134 20.11 GB 256 53.1% f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01 UN 10.234.72.140 122.25 KB 256 41.9% 501ea498-8fed-4cc8-a23a-c99492bc4f26 e07 UN 10.234.72.139 20.46 GB 256 40.2% 95661392-ccd8-4592-a76f-1c99f7cdf23a e07 UN 10.234.72.138 13.21 GB 256 40.9% 0d6725f0-1357-423d-85c1-153fb94257d5 e03 UN 10.234.72.137 13.34 GB 256 42.9% 492190d7-3055-4167-8699-9c6560e28164 e03 UN 10.234.72.136 14.16 GB 256 39.0% 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01 Datacenter: s1 ============== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.98.255.139 19.19 GB 256 43.8% 3523e80c-8468-4502-b334-79eabc3357f0 g10 UN 10.98.255.138 14.9 GB 256 42.4% a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02 UN 10.98.255.137 12.49 GB 256 38.4% f851b6ee-f1e4-431b-8beb-e7b173a77342 i02 UN 10.98.255.136 14.13 GB 256 42.9% 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09 UN 10.98.255.135 12.16 GB 256 40.4% e2d020a5-97a9-48d4-870c-d10b59858763 a09 UN 10.98.255.134 18.85 GB 256 52.3% 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06 UN 10.98.255.140 2.24 GB 256 39.9% 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10 We can see that the 2nd new node (10.234.72.140) contains only 122,25KB. The new node in the 1st datacenter contains now 2,24 GB because we were inserting data in the cluster while adding the new nodes. Then we started a repair from the new node in the 2nd datacenter : time nodetool repair We can see that the old nodes are sending data to the new node : root@node007:~# nodetool netstats Mode: NORMAL Not sending any streams. Streaming from: /10.98.255.137 hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db sections=130 progress=0/15598366 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db sections=107 progress=0/429517 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db sections=109 progress=0/696057 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db sections=57 progress=0/189844 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db sections=124 progress=56492032/4597955 - 1228% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db sections=113 progress=0/850632 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db sections=130 progress=0/16815423 - 0% Streaming from: /10.98.255.136 hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db sections=104 progress=0/3545520 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db sections=43 progress=0/252976 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db sections=91 progress=0/443365 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db sections=101 progress=11337728/4188624 - 270% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db sections=94 progress=0/758418 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db sections=88 progress=0/396552 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db sections=105 progress=0/12363907 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db sections=87 progress=0/506537 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db sections=42 progress=0/197531 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db sections=110 progress=0/12689898 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db sections=83 progress=0/379628 - 0% Streaming from: /10.98.255.134 hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db sections=108 progress=0/10715348 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db sections=82 progress=0/8177161 - 0% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db sections=84 progress=25613959168/8556942 - 299335% hbxtest: /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db sections=79 progress=0/506293 - 0% Pool Name Active Pending Completed Commands n/a 0 835 Responses n/a 0 6765 So, it looks that the repair is required if we want to add new nodes in our platform, but I don't understand why. Regards. Jean Armel
