On Tue, Oct 29, 2013 at 11:45 AM, Narendra Sharma <narendra.sha...@gmail.com > wrote:
> We had a cluster of 4 nodes in AWS. The average load on each node was > approx 750GB. We added 4 new nodes. It is now more than 30 hours and the > node is still in JOINING mode. > Specifically I am analyzing the one with IP 10.3.1.29. There is no > compaction or streaming or index building happening. > If your cluster has RF>2, you are bootstrapping two nodes into the same range simultaneously. That is not supported. [1,2] The node you are having the problem with is in the range that is probably overlapping. If I were you I would : 1) stop all "Joining" nodes and wipe their state including system keyspace 2) optionally "removetoken" any nodes which remain in cluster gossip state after stopping 3) re-start/bootstrap them one at a time, waiting for each to complete bootstrapping before starting the next one 4) (unrelated) Upgrade from 1.1.6 to the head of 1.1.x ASAP. =Rob [1] https://issues.apache.org/jira/browse/CASSANDRA-2434 [2] https://issues.apache.org/jira/browse/CASSANDRA-2434?focusedCommentId=13091851&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13091851
