Hi Aaron, I tried again to add a node in the cluster. This time, I added the new node in the seeds list after the bootstrap (the first time, I added the new node in the seeds list before the bootstrap).
And it works !!! Thanks Aaron. Regards. Jean Armel. 2013/2/22 Jean-Armel Luce <jaluc...@gmail.com> > Thanks Aaron. > > I shall investigate more next week about this. > > regards. > > Jean Armel > > > 2013/2/22 aaron morton <aa...@thelastpickle.com> > >> So, it looks that the repair is required if we want to add new nodes in >> our platform, but I don't understand why. >> >> Bootstrapping should take care of it. But new seed nodes do not >> bootstrap. Check the logs on the nodes you added to see what messages have >> "bootstrap" in them. >> >> Anytime you are worried about things like this throw in a nodetool >> repair. If you are using QUOURM for read and writes you will still be >> getting consistent data, so long as you have only added one node. Or one >> node every RF'th nodes. >> >> Cheers >> >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Developer >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 22/02/2013, at 9:55 PM, Jean-Armel Luce <jaluc...@gmail.com> wrote: >> >> Hi Aaron, >> >> Thanks for your answer. >> >> >> I apologize, I did a mistake in my 1st mail. The cluster was only 12 >> nodes instead of 16 (it is a test cluster). >> There are 2 datacenters b1 and s1. >> >> Here is the result of nodetool status after adding a new node in the 1st >> datacenter (dc s1): >> root@node007:~# nodetool status >> Datacenter: b1 >> ============== >> Status=Up/Down >> |/ State=Normal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host >> ID Rack >> UN 10.234.72.135 10.71 GB 256 44.6% >> 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01 >> UN 10.234.72.134 16.74 GB 256 63.7% >> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01 >> UN 10.234.72.139 17.09 GB 256 62.0% >> 95661392-ccd8-4592-a76f-1c99f7cdf23a e07 >> UN 10.234.72.138 10.96 GB 256 42.9% >> 0d6725f0-1357-423d-85c1-153fb94257d5 e03 >> UN 10.234.72.137 11.09 GB 256 45.7% >> 492190d7-3055-4167-8699-9c6560e28164 e03 >> UN 10.234.72.136 11.91 GB 256 41.1% >> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01 >> Datacenter: s1 >> ============== >> Status=Up/Down >> |/ State=Normal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host >> ID Rack >> UN 10.98.255.139 16.94 GB 256 43.8% >> 3523e80c-8468-4502-b334-79eabc3357f0 g10 >> UN 10.98.255.138 12.62 GB 256 42.4% >> a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02 >> UN 10.98.255.137 10.59 GB 256 38.4% >> f851b6ee-f1e4-431b-8beb-e7b173a77342 i02 >> UN 10.98.255.136 11.89 GB 256 42.9% >> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09 >> UN 10.98.255.135 10.29 GB 256 40.4% >> e2d020a5-97a9-48d4-870c-d10b59858763 a09 >> UN 10.98.255.134 16.19 GB 256 52.3% >> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06 >> UN 10.98.255.140 127.84 KB 256 39.9% >> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10 >> >> We can see that the new node (10.98.255.140) contains only 127,84KB. >> We saw also that there was no network traffic between the nodes. >> >> Then we added a new node in the 2nd datacenter (dc b1) >> >> >> >> root@node007:~# nodetool status >> Datacenter: b1 >> ============== >> Status=Up/Down >> |/ State=Normal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host >> ID Rack >> UN 10.234.72.135 12.95 GB 256 42.0% >> 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01 >> UN 10.234.72.134 20.11 GB 256 53.1% >> f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01 >> UN 10.234.72.140 122.25 KB 256 41.9% >> 501ea498-8fed-4cc8-a23a-c99492bc4f26 e07 >> UN 10.234.72.139 20.46 GB 256 40.2% >> 95661392-ccd8-4592-a76f-1c99f7cdf23a e07 >> UN 10.234.72.138 13.21 GB 256 40.9% >> 0d6725f0-1357-423d-85c1-153fb94257d5 e03 >> UN 10.234.72.137 13.34 GB 256 42.9% >> 492190d7-3055-4167-8699-9c6560e28164 e03 >> UN 10.234.72.136 14.16 GB 256 39.0% >> 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01 >> Datacenter: s1 >> ============== >> Status=Up/Down >> |/ State=Normal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host >> ID Rack >> UN 10.98.255.139 19.19 GB 256 43.8% >> 3523e80c-8468-4502-b334-79eabc3357f0 g10 >> UN 10.98.255.138 14.9 GB 256 42.4% >> a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02 >> UN 10.98.255.137 12.49 GB 256 38.4% >> f851b6ee-f1e4-431b-8beb-e7b173a77342 i02 >> UN 10.98.255.136 14.13 GB 256 42.9% >> 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09 >> UN 10.98.255.135 12.16 GB 256 40.4% >> e2d020a5-97a9-48d4-870c-d10b59858763 a09 >> UN 10.98.255.134 18.85 GB 256 52.3% >> 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06 >> UN 10.98.255.140 2.24 GB 256 39.9% >> 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10 >> >> >> We can see that the 2nd new node (10.234.72.140) contains only 122,25KB. >> The new node in the 1st datacenter contains now 2,24 GB because we were >> inserting data in the cluster while adding the new nodes. >> >> Then we started a repair from the new node in the 2nd datacenter : >> time nodetool repair >> >> >> We can see that the old nodes are sending data to the new node : >> root@node007:~# nodetool netstats >> Mode: NORMAL >> Not sending any streams. >> Streaming from: /10.98.255.137 >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db >> sections=130 progress=0/15598366 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db >> sections=107 progress=0/429517 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db >> sections=109 progress=0/696057 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db >> sections=57 progress=0/189844 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db >> sections=124 progress=56492032/4597955 - 1228% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db >> sections=113 progress=0/850632 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db >> sections=130 progress=0/16815423 - 0% >> Streaming from: /10.98.255.136 >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db >> sections=104 progress=0/3545520 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db >> sections=43 progress=0/252976 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db >> sections=91 progress=0/443365 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db >> sections=101 progress=11337728/4188624 - 270% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db >> sections=94 progress=0/758418 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db >> sections=88 progress=0/396552 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db >> sections=105 progress=0/12363907 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db >> sections=87 progress=0/506537 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db >> sections=42 progress=0/197531 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db >> sections=110 progress=0/12689898 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db >> sections=83 progress=0/379628 - 0% >> Streaming from: /10.98.255.134 >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db >> sections=108 progress=0/10715348 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db >> sections=82 progress=0/8177161 - 0% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db >> sections=84 progress=25613959168/8556942 - 299335% >> hbxtest: >> /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db >> sections=79 progress=0/506293 - 0% >> Pool Name Active Pending Completed >> Commands n/a 0 835 >> Responses n/a 0 6765 >> >> So, it looks that the repair is required if we want to add new nodes in >> our platform, but I don't understand why. >> >> Regards. >> >> Jean Armel >> >> >> >> >
