I'm trying to join node to the ring with nodetool join command but it fails
with error message about inconsistent replica.
My steps:
1. run cassandra with -Dcassandra.join_ring=false
2. wait couple of minutes
3. ensure that all nodes are in UN state
4. run nodetool join
Joining fails with message:
error: A node required to move the data consistently is down
(/XX.XX.XX.XX). If you wish to move the data from a potentially
inconsistent replica, restart the node with
-Dcassandra.consistent.rangemovement
=false
-- StackTrace --
java.lang.RuntimeException: A node required to move the data consistently
is down (/XX.XX.X.XX). If you wish to move the data from a potentially
inconsistent replica, restart the node with -Dcassandra.con
sistent.rangemovement=false
at
org.apache.cassandra.dht.RangeStreamer.getAllRangesWithStrictSourcesFor(RangeStreamer.java:275)
at
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:158)
at
org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:83)
...
I'm pretty sure that node XX.XX.XX.XX is up and reachable over network.
Also no references in log that XX.XX.XX.XX node is went down.
Different tries shows different nodes are "down", but all nodes are from
remote data centers.
Log from XX.XX.XX.XX node:
DEBUG [GossipStage:1] 2016-12-09 16:37:41,557 StorageService.java:1893 -
Node /YY.YY.YY.YY state bootstrapping, token [-1853578429238772836,
-155879016060746037, 8986645362194256101, 7352444016819915166,
-7093258492559265403, ..., -5513577076972264694, 411791028346144716,
-7405822878444068495]
DEBUG [PendingRangeCalculator:1] 2016-12-09 16:37:47,225
PendingRangeCalculatorService.java:66 - finished calculation for 8
keyspaces in 5667ms
C* v3.0.9
Any clues how to troubleshoot it?
