I added a second node to a single-node ring. RF=1. I can't get the new
node to receive any data. Logs look fine. Here's what nodetool reports:
# nodetool -h localhost ring
Address DC Rack Status State Load
Owns Token
85070591730234615865843651857942052864
10.102.37.168 datacenter1 rack1 Up Normal 807.81 GB
50.00% 0
10.80.161.101 datacenter1 rack1 Up Normal 1.15 MB
50.00% 85070591730234615865843651857942052864
Just a "little" imbalance. Yes, I use partitioner:
org.apache.cassandra.dht.RandomPartitioner
I tried moving the new node's token up/down by 1 and it triggers the log
messages you'd expect, but no data gets transferred. How do I
troubleshoot this? Below are the log messages I see when restarting the
new node:
INFO [main] 2012-03-15 17:31:08,616 AbstractCassandraDaemon.java (line
120) JVM vendor/version:
Java HotSpot(TM) 64-Bit Server VM/1.6.0_24
INFO [main] 2012-03-15 17:31:14,812 CommitLog.java (line 178) Log
replay complete, 8 replayed mutations
INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 390)
Cassandra version: 1.0.6
INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 391)
Thrift API version: 19.19.0
INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 404)
Loading persisted ring state
INFO [main] 2012-03-15 17:31:14,834 StorageService.java (line 482)
Starting up server gossip
INFO [main] 2012-03-15 17:31:15,372 MessagingService.java (line 247)
Starting Encrypted Messaging Service on SSL port 7000
INFO [main] 2012-03-15 17:31:15,376 MessagingService.java (line 268)
Starting Messaging Service on port 7001
INFO [main] 2012-03-15 17:31:15,401 StorageService.java (line 579)
Using saved token 85070591730234615865843651857942052864
INFO [main] 2012-03-15 17:31:15,402 ColumnFamilyStore.java (line 692)
Enqueuing flush of Memtable-LocationInfo@645492252(53/66 serialized/live
bytes, 2 ops)
INFO [FlushWriter:1] 2012-03-15 17:31:15,403 Memtable.java (line 240)
Writing Memtable-LocationInfo@645492252(53/66 serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2012-03-15 17:31:15,421 Memtable.java (line 277)
Completed flushing /mnt/ebs/data/system/LocationInfo-hc-32-Data.db (163
bytes)
INFO [main] 2012-03-15 17:31:15,424 StorageService.java (line 948) Node
/10.80.161.101 state jump to normal
INFO [main] 2012-03-15 17:31:15,434 StorageService.java (line 589)
Bootstrap/Replace/Move completed! Now serving reads.
# describe keyspace
Keyspace: rslog_production:
Replication Strategy: org.apache.cassandra.locator.SimpleStrategy
Durable Writes: true
Options: [replication_factor:1]
Column Families:
