Just checking if this sorted it's self out? Cheers
----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 10/02/2013, at 1:15 AM, Jouni Hartikainen <jouni.hartikai...@reaktor.fi> wrote: > Hello all, > > I have a cluster of three nodes running 1.2.1 and I'd like to increase the > capacity by adding a new node. I'm using virtual nodes with 256 tokens and > planning to use the same configuration for the new node as well. > > My cluster looks like this before adding the new node: > > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 192.168.154.11 1.49 GB 256 100.0% > 234b82a4-3812-4261-adab-deb805942d63 rack1 > UN 192.168.154.12 1.6 GB 256 100.0% > 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1 > UN 192.168.154.13 1.64 GB 256 100.0% > 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1 > > > And corresponding gossipinfo: > > /192.168.154.12 > RPC_ADDRESS:192.168.154.12 > DC:datacenter1 > STATUS:NORMAL,-1072164398478041156 > LOAD:1.719425018E9 > SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d > NET_VERSION:6 > HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6 > SEVERITY:0.0 > RELEASE_VERSION:1.2.1 > RACK:rack1 > /192.168.154.11 > RPC_ADDRESS:192.168.154.11 > DC:datacenter1 > STATUS:NORMAL,-1158837144480089281 > LOAD:1.514343678E9 > SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d > NET_VERSION:6 > HOST_ID:234b82a4-3812-4261-adab-deb805942d63 > SEVERITY:0.0 > RELEASE_VERSION:1.2.1 > RACK:rack1 > /192.168.154.13 > RPC_ADDRESS:192.168.154.13 > DC:datacenter1 > STATUS:NORMAL,-1135137292201587328 > LOAD:1.765093695E9 > SCHEMA:ef2c294e-1a74-32c1-b169-3a6465b2053d > NET_VERSION:6 > HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff > SEVERITY:0.0 > RELEASE_VERSION:1.2.1 > RACK:rack1 > > > I have now set the correct net addresses & seeds in the cassandra.yaml of the > new node (.14) and then started it with num_tokens set to 256 and > initial_token commented out. Everything seems to go OK as I get the following > prints on the log: > > On node 192.168.154.11: > > INFO [GossipStage:1] 2013-02-09 12:30:28,126 Gossiper.java (line 784) Node > /192.168.154.14 is now part of the cluster > INFO [GossipStage:1] 2013-02-09 12:30:28,128 Gossiper.java (line 750) > InetAddress /192.168.154.14 is now UP > INFO [MiscStage:1] 2013-02-09 12:30:59,255 StreamOut.java (line 114) > Beginning transfer to /192.168.154.14 > > And on node 192.168.154.14 (the new node): > > INFO 12:30:26,843 Loading persisted ring state > INFO 12:30:26,846 Starting up server gossip > WARN 12:30:26,853 No host ID found, created > a4a0b918-a1c8-4acc-a050-672a96a5f110 (Note: This should happen exactly once > per node). > INFO 12:30:26,979 Starting Messaging Service on port 7000 > INFO 12:30:27,014 JOINING: waiting for ring information > INFO 12:30:28,602 Node /192.168.154.11 is now part of the cluster > INFO 12:30:28,603 InetAddress /192.168.154.11 is now UP > INFO 12:30:28,675 Node /192.168.154.12 is now part of the cluster > INFO 12:30:28,678 InetAddress /192.168.154.12 is now UP > INFO 12:30:28,751 Node /192.168.154.13 is now part of the cluster > INFO 12:30:28,751 InetAddress /192.168.154.13 is now UP > INFO 12:30:29,015 JOINING: schema complete, ready to bootstrap > INFO 12:30:29,015 JOINING: getting bootstrap token > INFO 12:30:29,157 JOINING: sleeping 30000 ms for pending range setup > INFO 12:30:59,159 JOINING: Starting to bootstrap... > > However, the new node does not show up in "nodetool status" (even if queried > from the new node itself): > > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN 192.168.154.11 1.49 GB 256 100.0% > 234b82a4-3812-4261-adab-deb805942d63 rack1 > UN 192.168.154.12 1.6 GB 256 100.0% > 577db21e-81ef-45fd-a67b-cfd39455c0f6 rack1 > UN 192.168.154.13 1.64 GB 256 100.0% > 6187cc5d-d44c-45cb-b738-1b87f5ae3dff rack1 > > It shows up in the gossip still: > > /192.168.154.12 > RPC_ADDRESS:192.168.154.12 > DC:datacenter1 > STATUS:NORMAL,-1072164398478041156 > LOAD:1.719430632E9 > SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 > NET_VERSION:6 > HOST_ID:577db21e-81ef-45fd-a67b-cfd39455c0f6 > SEVERITY:0.0 > RELEASE_VERSION:1.2.1-SNAPSHOT > RACK:rack1 > /192.168.154.14 > RPC_ADDRESS:192.168.154.14 > DC:datacenter1 > STATUS:BOOT,8077752099299332137 > LOAD:105101.0 > SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 > NET_VERSION:6 > HOST_ID:a4a0b918-a1c8-4acc-a050-672a96a5f110 > RELEASE_VERSION:1.2.1-SNAPSHOT > RACK:rack1 > /192.168.154.11 > RPC_ADDRESS:192.168.154.11 > DC:datacenter1 > STATUS:NORMAL,-1158837144480089281 > LOAD:1.596505929E9 > SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 > NET_VERSION:6 > HOST_ID:234b82a4-3812-4261-adab-deb805942d63 > SEVERITY:0.0 > RELEASE_VERSION:1.2.1-SNAPSHOT > RACK:rack1 > /192.168.154.13 > RPC_ADDRESS:192.168.154.13 > DC:datacenter1 > STATUS:NORMAL,-1135137292201587328 > LOAD:1.765062944E9 > SCHEMA:19657c82-a7eb-37a8-b436-0ea712c57db2 > NET_VERSION:6 > HOST_ID:6187cc5d-d44c-45cb-b738-1b87f5ae3dff > SEVERITY:-6.776263578034403E-21 > RELEASE_VERSION:1.2.1-SNAPSHOT > RACK:rack1 > > > I also noticed that the new node is not listening for its thrift port (9160) > until all data streaming has ended. > > > Couple of questions after all this: > 1) Is the new node not showing up in the "nodetool status" expected behavior? > 2) Do I have to wait for all the data streaming to end until the new node > starts to accept writes? If yes, has this changed from 1.1? (I remember nodes > becoming writeable immediately after booting, but readable only after > streaming has ended) > 3) "nodetool info" on the new node raises an exception "Exception in thread > "main" java.lang.AssertionError at > org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:465)" > This is probably not normal? > 4) Is it so that the virtual tokens for the new node are not assigned before > the data streaming ends? ("select * from peers;" on .11 returns null for the > tokens of .14) If so, how the existing nodes decide what data to stream to a > new node? > > > -Jouni
