ahh, I think you may have hit a corner case here. If the RF still 1 ?
> INFO [AntiEntropySessions:1] 2012-03-16 06:15:13,727 > AntiEntropyService.java (line 663) [repair #%s] No neighbors to repair > with on range %s: session completed Means there are no nodes which share the range with this node. So there is nothing to repair. To put it another way: As far is 161.101 is concerned none of the keys it is responsible for are stored on another node. So there are no other nodes that could be involved in a repair session. It looks like some data may have been written to 161.101 so I think the safest approach would be: * increase the RF to 2 * repair * decrease the RF to 1 When you added the node was auto_bootstrap enabled ? I would have thought that would stream data from the first node to the new one. Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 16/03/2012, at 7:22 PM, Thorsten von Eicken wrote: > Thanks for the suggestion Aaron, unfortunately, that seems to do > absolutely nothing: > > # nodetool -h localhost repair > INFO [RMI TCP Connection(160)-127.0.0.1] 2012-03-16 06:15:13,718 > StorageService.java (line 1770) Starting repair command #1, repairing 1 > ranges. > INFO [AntiEntropySessions:1] 2012-03-16 06:15:13,727 > AntiEntropyService.java (line 658) [repair > #6472b290-6f2f-11e1-0000-472739b10cff] new session: will sync > /10.80.161.101 on range (0,85070591730234615865843651857942052864] for > rslog_production.[users, req_text, req_attr_idx, req_word_idx, > req_word_freq, sessions, requests, info] > INFO [AntiEntropySessions:1] 2012-03-16 06:15:13,727 > AntiEntropyService.java (line 663) [repair #%s] No neighbors to repair > with on range %s: session completed > INFO [RMI TCP Connection(160)-127.0.0.1] 2012-03-16 06:15:13,727 > StorageService.java (line 1807) Repair command #1 completed successfully > > Stumped... > TvE > > > On 3/15/2012 6:41 PM, aaron morton wrote: >> trying running nodetool repair on 10.80.161.101 and then cleanup >> on 10.102.37.168 if everything is ok. >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >> >> On 16/03/2012, at 6:45 AM, Thorsten von Eicken wrote: >> >>> I added a second node to a single-node ring. RF=1. I can't get the new >>> node to receive any data. Logs look fine. Here's what nodetool reports: >>> >>> # nodetool -h localhost ring >>> Address DC Rack Status State Load >>> Owns Token >>> >>> 85070591730234615865843651857942052864 >>> 10.102.37.168 datacenter1 rack1 Up Normal 807.81 GB >>> 50.00% 0 >>> 10.80.161.101 datacenter1 rack1 Up Normal 1.15 MB >>> 50.00% 85070591730234615865843651857942052864 >>> >>> Just a "little" imbalance. Yes, I use partitioner: >>> org.apache.cassandra.dht.RandomPartitioner >>> I tried moving the new node's token up/down by 1 and it triggers the log >>> messages you'd expect, but no data gets transferred. How do I >>> troubleshoot this? Below are the log messages I see when restarting the >>> new node: >>> >>> INFO [main] 2012-03-15 17:31:08,616 AbstractCassandraDaemon.java (line >>> 120) JVM vendor/version: >>> Java HotSpot(TM) 64-Bit Server VM/1.6.0_24 >>> INFO [main] 2012-03-15 17:31:14,812 CommitLog.java (line 178) Log >>> replay complete, 8 replayed mutations >>> INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 390) >>> Cassandra version: 1.0.6 >>> INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 391) >>> Thrift API version: 19.19.0 >>> INFO [main] 2012-03-15 17:31:14,825 StorageService.java (line 404) >>> Loading persisted ring state >>> INFO [main] 2012-03-15 17:31:14,834 StorageService.java (line 482) >>> Starting up server gossip >>> INFO [main] 2012-03-15 17:31:15,372 MessagingService.java (line 247) >>> Starting Encrypted Messaging Service on SSL port 7000 >>> INFO [main] 2012-03-15 17:31:15,376 MessagingService.java (line 268) >>> Starting Messaging Service on port 7001 >>> INFO [main] 2012-03-15 17:31:15,401 StorageService.java (line 579) >>> Using saved token 85070591730234615865843651857942052864 >>> INFO [main] 2012-03-15 17:31:15,402 ColumnFamilyStore.java (line 692) >>> Enqueuing flush of Memtable-LocationInfo@645492252(53/66 serialized/live >>> bytes, 2 ops) >>> INFO [FlushWriter:1] 2012-03-15 17:31:15,403 Memtable.java (line 240) >>> Writing Memtable-LocationInfo@645492252(53/66 serialized/live bytes, >>> 2 ops) >>> INFO [FlushWriter:1] 2012-03-15 17:31:15,421 Memtable.java (line 277) >>> Completed flushing /mnt/ebs/data/system/LocationInfo-hc-32-Data.db (163 >>> bytes) >>> INFO [main] 2012-03-15 17:31:15,424 StorageService.java (line 948) Node >>> /10.80.161.101 state jump to normal >>> INFO [main] 2012-03-15 17:31:15,434 StorageService.java (line 589) >>> Bootstrap/Replace/Move completed! Now serving reads. >>> >>> # describe keyspace >>> Keyspace: rslog_production: >>> Replication Strategy: org.apache.cassandra.locator.SimpleStrategy >>> Durable Writes: true >>> Options: [replication_factor:1] >>> Column Families: >>> >>
