Hi All
I saw something curious when I restarted our cluster following a full shutdown
whereby the result of calling nodetool getendpoints showed inconsistent results
as each node was started in turn.
Prior to shutdown, we had the following:
$ nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.0.3.11 46.53 GiB 256 51.1%
35dcbc33-efb3-4c98-9063-aa451e6e0c7f rack1
UN 10.0.3.12 44.44 GiB 256 48.9%
8c13e94a-2c8e-43a3-9fb0-11d4b5b4606f rack1
UN 10.0.3.21 41.42 GiB 256 45.4%
6b619cc8-e398-4e48-8542-09d84c3710f6 rack2
UN 10.0.3.22 49.55 GiB 256 54.6%
962b04ee-e135-4ed5-87f9-7463c0b18860 rack2
UN 10.0.3.31 45.76 GiB 256 50.2%
2a90ac41-a741-472f-8c74-a382a24eb65f rack3
UN 10.0.3.32 45.2 GiB 256 49.8%
e7dc5e68-bbde-4f6b-9426-43cf13c679c8 rack3
$ nodetool getendpoints ospreypro Users 39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.12
10.0.3.22
10.0.3.32
As each node was started up, the results differed as follows:
administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users
39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.11
administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users
39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.11
10.0.3.12
administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users
39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.11
10.0.3.12
10.0.3.21
administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users
39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.11
10.0.3.12
10.0.3.22
administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users
39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.12
10.0.3.22
10.0.3.31
administrator@OspreyProSRV1-R1N1:~$ nodetool getendpoints ospreypro Users
39befa1f-dee1-48c4-9702-e8e8332b84df
10.0.3.12
10.0.3.22
10.0.3.32
Eventually, the endpoints settles on the expected values.
Seeing as the token ranges are stored in system.peers (stored on each node),
why would I see different endpoints as I do?
Regards
Marcel Villet
Osprey Pro Team Leader and Lead Architect
CT LAB (PTY) LTD - Bringing Quality To Power
T +27 (0)21-880-9915 | F +27 (0)21-880-1088 |
www.ctlab.com<http://www.ctlab.com/> |
www.pq-portal.com<http://www.pq-portal.com/rdPage.aspx>
15 Termo Lane, Techno Park, Stellenbosch, South Africa, 7600 | PO Box 897,
Stellenbosch,7599.
[cid:E9CDFBB1-258E-48CA-8E20-3ED934168E78]
This email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed. If
you have received this email in error please notify the system
manager:administra...@ctlab.com
