Hello all,
I have a cluster of three nodes running 1.2.1 and I'd like to increase the
capacity by adding a new node. I'm using virtual nodes with 256 tokens and
planning to use the same configuration for the new node as well.
My cluster looks like this before adding the new node:
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 192.168.154.11 1.49 GB 256 100.0%
234b82a4-3812-4261-adab-deb805942d63 rack1
UN 192.168.154.12 1.6 GB 256 100.0%
577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1
UN 192.168.154.13 1.64 GB 256 100.0%
6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1
And corresponding gossipinfo:
/192.168.154.12
RPC_ADDRESS:192.168.154.12
DC:datacenter1
STATUS:NORMAL,-1072164398478041156
LOAD:1.719425018E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
/192.168.154.11
RPC_ADDRESS:192.168.154.11
DC:datacenter1
STATUS:NORMAL,-1158837144480089281
LOAD:1.514343678E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:234b82a4-3812-4261-adab-deb805942d63
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
/192.168.154.13
RPC_ADDRESS:192.168.154.13
DC:datacenter1
STATUS:NORMAL,-1135137292201587328
LOAD:1.765093695E9
SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d
NET_VERSION:6
HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
SEVERITY:0.0
RELEASE_VERSION:1.2.1
RACK:rack1
I have now set the correct net addresses & seeds in the cassandra.yaml of the
new node (.14) and then started it with num_tokens set to 256 and initial_token
commented out. Everything seems to go OK as I get the following prints on the
log:
On node 192.168.154.11:
INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node
/192.168.154.14 is now part of the cluster
INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750)
InetAddress /192.168.154.14 is now UP
INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) Beginning
transfer to /192.168.154.14
And on node 192.168.154.14 (the new node):
INFO 12:30:26,843 Loading persisted ring state
INFO 12:30:26,846 Starting up server gossip
WARN 12:30:26,853 No host ID found, created
a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note: This should happen exactly once per
node).
INFO 12:30:26,979 Starting Messaging Service on port 7000
INFO 12:30:27,014 JOINING: waiting for ring information
INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster
INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP
INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster
INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP
INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster
INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP
INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap
INFO 12:30:29,015 JOINING: getting bootstrap token
INFO 12:30:29,157 JOINING: sleeping 30000 ms for pending range setup
INFO 12:30:59,159 JOINING: Starting to bootstrap...
However, the new node does not show up in "nodetool status" (even if queried
from the new node itself):
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 192.168.154.11 1.49 GB 256 100.0%
234b82a4-3812-4261-adab-deb805942d63 rack1
UN 192.168.154.12 1.6 GB 256 100.0%
577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1
UN 192.168.154.13 1.64 GB 256 100.0%
6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1
It shows up in the gossip still:
/192.168.154.12
RPC_ADDRESS:192.168.154.12
DC:datacenter1
STATUS:NORMAL,-1072164398478041156
LOAD:1.719430632E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6
SEVERITY:0.0
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.14
RPC_ADDRESS:192.168.154.14
DC:datacenter1
STATUS:BOOT,8077752099299332137
LOAD:105101.0
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.11
RPC_ADDRESS:192.168.154.11
DC:datacenter1
STATUS:NORMAL,-1158837144480089281
LOAD:1.596505929E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:234b82a4-3812-4261-adab-deb805942d63
SEVERITY:0.0
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
/192.168.154.13
RPC_ADDRESS:192.168.154.13
DC:datacenter1
STATUS:NORMAL,-1135137292201587328
LOAD:1.765062944E9
SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2
NET_VERSION:6
HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff
SEVERITY:-6.776263578034403E-21
RELEASE_VERSION:1.2.1-SNAPSHOT
RACK:rack1
I also noticed that the new node is not listening for its thrift port (9160)
until all data streaming has ended.
Couple of questions after all this:
1) Is the new node not showing up in the "nodetool status" expected behavior?
2) Do I have to wait for all the data streaming to end until the new node
starts to accept writes? If yes, has this changed from 1.1? (I remember nodes
becoming writeable immediately after booting, but readable only after streaming
has ended)
3) "nodetool info" on the new node raises an exception "Exception in thread
"main" java.lang.AssertionError at
org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)"
This is probably not normal?
4) Is it so that the virtual tokens for the new node are not assigned before
the data streaming ends? ("select * from peers;" on .11 returns null for the
tokens of .14) If so, how the existing nodes decide what data to stream to a
new node?
-Jouni
