Unfortunately not, I've moved on to trying to add the nodes the current
cluster and then decommission the "old" ones.
But even that is not working, this is the strangest of things : while
trying to add a new node, I
- set its token to an existing value+1
- ensure the yaml (clutser name, partitionner, etc) are the same
- verified I can connect to the :7000 ports across machines
- cleared the data and commitlog directory
when I start the node, it goes through the bootstrap process but never
"imports" the schema from the cluster (the data/ directory stays empty,
permissions are correct) and I get errors when some reads come in after the
bootstrap completes. I've tried restarting the node with -D replace.token
The log is below. I've been at it all day so I've been adding the node &
decommissioning it, assuming that would clear any state in the cluster.
INFO [main] 2013-05-03 21:10:22,126 StorageService.java (line 788)
JOINING: waiting for ring information
INFO [GossipStage:1] 2013-05-03 21:10:26,421 Gossiper.java (line 838)
InetAddress /172.16.0.42 is now dead.
INFO [GossipStage:1] 2013-05-03 21:10:26,427 StorageService.java (line
1303) Removing token Token(bytes[03abaa3000000000000001]) for /172.16.0.42
INFO [GossipStage:1] 2013-05-03 21:10:26,429 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@444462602(47/58
serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:26,430 Memtable.java (line 264)
Writing Memtable-LocationInfo@444462602(47/58 serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:26,542 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db
(160 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=769)
INFO [GossipStage:1] 2013-05-03 21:10:26,544 Gossiper.java (line 858) Node
/{other_node_A} is now part of the cluster
INFO [GossipStage:1] 2013-05-03 21:10:26,545 Gossiper.java (line 824)
InetAddress /{other_node_A} is now UP
INFO [GossipStage:1] 2013-05-03 21:10:26,547 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@1290155526(30/37
serialized/live bytes, 1 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:26,548 Memtable.java (line 264)
Writing Memtable-LocationInfo@1290155526(30/37 serialized/live bytes, 1 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:26,662 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=862)
INFO [GossipStage:1] 2013-05-03 21:10:26,664 Gossiper.java (line 858) Node
/{other_node_B} is now part of the cluster
INFO [GossipStage:1] 2013-05-03 21:10:26,665 Gossiper.java (line 824)
InetAddress /{other_node_B} is now UP
INFO [GossipStage:1] 2013-05-03 21:10:26,666 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@899933215(30/37
serialized/live bytes, 1 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:26,667 Memtable.java (line 264)
Writing Memtable-LocationInfo@899933215(30/37 serialized/live bytes, 1 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:26,779 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=955)
INFO [GossipStage:1] 2013-05-03 21:10:26,781 Gossiper.java (line 858) Node
/{other_node_C} is now part of the cluster
INFO [GossipStage:1] 2013-05-03 21:10:26,782 Gossiper.java (line 824)
InetAddress /{other_node_C} is now UP
INFO [GossipStage:1] 2013-05-03 21:10:26,784 ColumnFamilyStore.java (line
674) Enqueuing flush of Memtable-LocationInfo@1542631196(30/37
serialized/live bytes, 1 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:26,785 Memtable.java (line 264)
Writing Memtable-LocationInfo@1542631196(30/37 serialized/live bytes, 1 ops)
INFO [CompactionExecutor:4] 2013-05-03 21:10:26,789 CompactionTask.java
(line 107) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-3-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-4-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-2-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-1-Data.db')]
INFO [FlushWriter:1] 2013-05-03 21:10:26,939 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db
(84 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1048)
INFO [GossipStage:1] 2013-05-03 21:10:26,942 Gossiper.java (line 838)
InetAddress /172.16.0.39 is now dead.
INFO [CompactionExecutor:4] 2013-05-03 21:10:26,986 CompactionTask.java
(line 232) Compacted to
[/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db,].
566 to 458 (~80% of original) bytes for 4 keys at 0.002287MB/s. Time:
191ms.
INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
JOINING: schema complete
INFO [main] 2013-05-03 21:10:52,130 StorageService.java (line 788)
JOINING: waiting for pending range calculation
INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
JOINING: calculation complete, ready to bootstrap
INFO [main] 2013-05-03 21:10:52,131 StorageService.java (line 788)
JOINING: getting bootstrap token
INFO [main] 2013-05-03 21:10:52,135 ColumnFamilyStore.java (line 674)
Enqueuing flush of Memtable-LocationInfo@1692466348(31/38 serialized/live
bytes, 1 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:52,136 Memtable.java (line 264)
Writing Memtable-LocationInfo@1692466348(31/38 serialized/live bytes, 1 ops)
INFO [FlushWriter:1] 2013-05-03 21:10:52,252 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db
(82 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1139)
INFO [main] 2013-05-03 21:10:52,254 StorageService.java (line 788)
JOINING: sleeping 30000 ms for pending range setup
INFO [main] 2013-05-03 21:11:22,255 StorageService.java (line 788)
JOINING: Starting to bootstrap...
INFO [main] 2013-05-03 21:11:22,272 ColumnFamilyStore.java (line 674)
Enqueuing flush of Memtable-LocationInfo@1458621244(48/60 serialized/live
bytes, 2 ops)
INFO [FlushWriter:1] 2013-05-03 21:11:22,273 Memtable.java (line 264)
Writing Memtable-LocationInfo@1458621244(48/60 serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2013-05-03 21:11:22,386 Memtable.java (line 305)
Completed flushing
/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db
(158 bytes) for commitlog position ReplayPosition(segmentId=1367608221854,
position=1315)
INFO [CompactionExecutor:8] 2013-05-03 21:11:22,389 CompactionTask.java
(line 107) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-8-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-5-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-7-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-6-Data.db')]
INFO [main] 2013-05-03 21:11:22,396 StorageService.java (line 1133) Node /
5.135.140.99 state jump to normal
INFO [main] 2013-05-03 21:11:22,397 StorageService.java (line 701)
Bootstrap/Replace/Move completed! Now serving reads.
INFO [main] 2013-05-03 21:11:22,415 CassandraDaemon.java (line 125)
Binding thrift service to /0.0.0.0:9160
INFO [main] 2013-05-03 21:11:22,417 CassandraDaemon.java (line 134) Using
TFastFramedTransport with a max frame size of 15728640 bytes.
INFO [main] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 161) Using
synchronous/threadpool thrift server on /0.0.0.0 : 9160
INFO [Thread-9] 2013-05-03 21:11:22,419 CassandraDaemon.java (line 213)
Listening for thrift clients...
INFO [CompactionExecutor:8] 2013-05-03 21:11:22,542 CompactionTask.java
(line 232) Compacted to
[/var/lib/cassandra/data/system/LocationInfo/system-LocationInfo-hf-9-Data.db,].
782 to 489 (~62% of original) bytes for 4 keys at 0.003088MB/s. Time:
151ms.
