Hello,
I have a a cluster with 12 nodes each in 2 datacenters for a total of 24
nodes.
I am attempting to add a 13th node in one of the datacenters. I have been
monitoring this process from the node itself with nodetool netstats and
from one of the existing nodes using nodetool status.
On the existing node I see the new node as UJ. I have watched the load
steadily climb up to about 203.4gb, and then over the last two hours it has
fluctuated a bit and has been steadily dropping to about 203.1gb
On the node that I am adding I watched over several hours as nodetool
netstats received data, however for the last couple of hours nodetool
netsats simply shows the ips of the other nodes in the cluster. It looks
something like to this...
Mode: JOINING
Bootstrap 659153b0-3ab6-11e5-8c94-5dd79366f3d9
/10.1.82.160
/10.1.82.162
/10.1.82.80
/10.2.123.74
/10.1.82.166
/10.1.82.158
/10.1.82.168
/10.1.82.150
/10.1.82.148
/10.2.123.2
/10.1.82.152
/10.1.82.156
/10.84.78.120
/10.2.123.80
/10.2.123.78
/10.81.122.64
/10.2.123.82
/10.2.123.84
/10.1.82.164
/10.81.122.62
/10.2.123.76
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed
Commands n/a 0 24
Responses n/a 0 1090793
So I'm trying to figure out... What is the node doing? Why is it still
joining? How long should I wait before being concerned?
Also...
The UUID next to the word 'Bootstrap' is NOT the host ID of the node
joining, it's actually the UUID of a different node already in the cluster.
This seems concerning to me, but again I'm not sure if this is expected
behavior or not.
ANY help would be greatly appreciated.
Thanks,
Stan
