> So, it looks that the repair is required if we want to add new nodes in our > platform, but I don't understand why. Bootstrapping should take care of it. But new seed nodes do not bootstrap. Check the logs on the nodes you added to see what messages have "bootstrap" in them.
Anytime you are worried about things like this throw in a nodetool repair. If you are using QUOURM for read and writes you will still be getting consistent data, so long as you have only added one node. Or one node every RF'th nodes. Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 22/02/2013, at 9:55 PM, Jean-Armel Luce <jaluc...@gmail.com> wrote: > Hi Aaron, > > Thanks for your answer. > > > I apologize, I did a mistake in my 1st mail. The cluster was only 12 nodes > instead of 16 (it is a test cluster). > There are 2 datacenters b1 and s1. > > Here is the result of nodetool status after adding a new node in the 1st > datacenter (dc s1): > root@node007:~# nodetool status > Datacenter: b1 > ============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 10.234.72.135 10.71 GB 256 44.6% > 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01 > UN 10.234.72.134 16.74 GB 256 63.7% > f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01 > UN 10.234.72.139 17.09 GB 256 62.0% > 95661392-ccd8-4592-a76f-1c99f7cdf23a e07 > UN 10.234.72.138 10.96 GB 256 42.9% > 0d6725f0-1357-423d-85c1-153fb94257d5 e03 > UN 10.234.72.137 11.09 GB 256 45.7% > 492190d7-3055-4167-8699-9c6560e28164 e03 > UN 10.234.72.136 11.91 GB 256 41.1% > 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01 > Datacenter: s1 > ============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 10.98.255.139 16.94 GB 256 43.8% > 3523e80c-8468-4502-b334-79eabc3357f0 g10 > UN 10.98.255.138 12.62 GB 256 42.4% > a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02 > UN 10.98.255.137 10.59 GB 256 38.4% > f851b6ee-f1e4-431b-8beb-e7b173a77342 i02 > UN 10.98.255.136 11.89 GB 256 42.9% > 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09 > UN 10.98.255.135 10.29 GB 256 40.4% > e2d020a5-97a9-48d4-870c-d10b59858763 a09 > UN 10.98.255.134 16.19 GB 256 52.3% > 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06 > UN 10.98.255.140 127.84 KB 256 39.9% > 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10 > > We can see that the new node (10.98.255.140) contains only 127,84KB. > We saw also that there was no network traffic between the nodes. > > Then we added a new node in the 2nd datacenter (dc b1) > > > > root@node007:~# nodetool status > Datacenter: b1 > ============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 10.234.72.135 12.95 GB 256 42.0% > 2fc583b2-822f-4347-9fab-5e9d10d548c9 c01 > UN 10.234.72.134 20.11 GB 256 53.1% > f209a8c5-7e1b-45b5-aa80-ed679bbbdbd1 e01 > UN 10.234.72.140 122.25 KB 256 41.9% > 501ea498-8fed-4cc8-a23a-c99492bc4f26 e07 > UN 10.234.72.139 20.46 GB 256 40.2% > 95661392-ccd8-4592-a76f-1c99f7cdf23a e07 > UN 10.234.72.138 13.21 GB 256 40.9% > 0d6725f0-1357-423d-85c1-153fb94257d5 e03 > UN 10.234.72.137 13.34 GB 256 42.9% > 492190d7-3055-4167-8699-9c6560e28164 e03 > UN 10.234.72.136 14.16 GB 256 39.0% > 3872f26c-5f2d-4fb3-9f5c-08b4c7762466 c01 > Datacenter: s1 > ============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 10.98.255.139 19.19 GB 256 43.8% > 3523e80c-8468-4502-b334-79eabc3357f0 g10 > UN 10.98.255.138 14.9 GB 256 42.4% > a2bcddf1-393e-453b-9d4f-9f7111c01d7f i02 > UN 10.98.255.137 12.49 GB 256 38.4% > f851b6ee-f1e4-431b-8beb-e7b173a77342 i02 > UN 10.98.255.136 14.13 GB 256 42.9% > 36fe902f-3fb1-4b6d-9e2c-71e601fa0f2e a09 > UN 10.98.255.135 12.16 GB 256 40.4% > e2d020a5-97a9-48d4-870c-d10b59858763 a09 > UN 10.98.255.134 18.85 GB 256 52.3% > 73e3376a-5a9f-4b8a-a119-c87ae1fafdcb h06 > UN 10.98.255.140 2.24 GB 256 39.9% > 3d5c33e6-35d0-40a0-b60d-2696fd5cbf72 g10 > > > We can see that the 2nd new node (10.234.72.140) contains only 122,25KB. > The new node in the 1st datacenter contains now 2,24 GB because we were > inserting data in the cluster while adding the new nodes. > > Then we started a repair from the new node in the 2nd datacenter : > time nodetool repair > > > We can see that the old nodes are sending data to the new node : > root@node007:~# nodetool netstats > Mode: NORMAL > Not sending any streams. > Streaming from: /10.98.255.137 > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-3-Data.db > sections=130 progress=0/15598366 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-198-Data.db > sections=107 progress=0/429517 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db > sections=109 progress=0/696057 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-119-Data.db > sections=57 progress=0/189844 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-199-Data.db > sections=124 progress=56492032/4597955 - 1228% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-196-Data.db > sections=113 progress=0/850632 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-16-Data.db > sections=130 progress=0/16815423 - 0% > Streaming from: /10.98.255.136 > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-17-Data.db > sections=104 progress=0/3545520 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db > sections=43 progress=0/252976 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db > sections=91 progress=0/443365 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-310-Data.db > sections=101 progress=11337728/4188624 - 270% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-307-Data.db > sections=94 progress=0/758418 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-309-Data.db > sections=88 progress=0/396552 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-4-Data.db > sections=105 progress=0/12363907 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db > sections=87 progress=0/506537 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-265-Data.db > sections=42 progress=0/197531 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-12-Data.db > sections=110 progress=0/12689898 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db > sections=83 progress=0/379628 - 0% > Streaming from: /10.98.255.134 > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-18-Data.db > sections=108 progress=0/10715348 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-2-Data.db > sections=82 progress=0/8177161 - 0% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-6-Data.db > sections=84 progress=25613959168/8556942 - 299335% > hbxtest: > /var/opt/hosting/db/iof/cassandra/data/hbxtest/medium_column/hbxtest-medium_column-ia-19-Data.db > sections=79 progress=0/506293 - 0% > Pool Name Active Pending Completed > Commands n/a 0 835 > Responses n/a 0 6765 > > So, it looks that the repair is required if we want to add new nodes in our > platform, but I don't understand why. > > Regards. > > Jean Armel > >
