I finally solved this by using nodetool move and specifying token number explicitly. Thanks for all your help !!
Cheers, Aishwarya On Tue, Aug 2, 2011 at 5:31 PM, aaron morton <aa...@thelastpickle.com> wrote: > initial_token is read from the yaml file once only, during bootstrap. It is > then stored in the LocationInfo system CF and used from there. > > It sounds like when you did the move you deleted these files, but then > started the nodes each with their own seed. So you created 3 separate > clusters, when each one bootstrapped it auto allocated itself an initial > token and stored it in LocationInfo. You have 3 clusters, each with one node > and each node has the same token as it was the first node into a new empty > cluster. > > Try this: > 1 - Shut down the two (lets call them B and C) you want to join the first > one (called A). > 2 - Delete their LocationInfo CF > 3 - ensure the seed list for ALL nodes points to node A. > 4 - ensure the initial token is set correctly for B and C > 5 - start B and C one at a time and make sure nodetool ring and describe > cluster; in the CLI agree before starting the next > > *IF* node A has the incorrect token I would fix this after you get B and C > back into the ring. > > Hope that helps. > > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > > On 3 Aug 2011, at 09:43, Aishwarya Venkataraman wrote: > >> I corrected the seed list and checked the cluster name. They are all >> good now. But still nodetool -ring shows only one node. >> >> INFO 21:36:59,735 Starting Messaging Service on port 7000 >> INFO 21:36:59,748 Using saved token 113427455640312814857969558651062452224 >> >> Nodes a_ipadrr and b_ipaddr have the same token >> 113427455640312814857969558651062452224. a_ipadrr is the new owner. >> >> >> All the nodes seem to be using the same initial token, despite me >> specifying an initial_token in the config file. Is this an issue ? How >> do I force cassandra to use the token in the cassandra.yaml file ? >> >> Thanks, >> Aishwarya >> >> >> On Tue, Aug 2, 2011 at 2:34 PM, Jonathan Ellis <jbel...@gmail.com> wrote: >>> Yes. >>> >>> Different cluster names could also cause this. >>> >>> On Tue, Aug 2, 2011 at 4:21 PM, Jeremiah Jordan >>> <jeremiah.jor...@morningstar.com> wrote: >>>> All of the nodes should have the same seedlist. Don't use localhost as >>>> one of the items in it if you have multiple nodes. >>>> >>>> On Tue, 2011-08-02 at 10:10 -0700, Aishwarya Venkataraman wrote: >>>>> Nodetool does not show me all the nodes. Assuming I have three nodes >>>>> A, B and C. The seedlist of A is localhost. Seedlist of B is >>>>> localhost, A_ipaddr and seedlist of C is localhost,B_ipaddr,A_ipaddr. >>>>> I have autobootstrap set to false for all 3 nodes since they all have >>>>> the correct data and do not hav to migrate data from any particular >>>>> node. >>>>> >>>>> My problem here is why does n't nodetool ring show me all nodes in the >>>>> ring ? I agree that the cluster thinks that only one node is present. >>>>> How do I fix this ? >>>>> >>>>> Thanks, >>>>> Aishwarya >>>>> >>>>> >>>>> On Tue, Aug 2, 2011 at 9:56 AM, samal <sa...@wakya.in> wrote: >>>>>> >>>>>> >>>>>>>> "ERROR 08:53:47,678 Internal error processing batch_mutate >>>>>>>> java.lang.IllegalStateException: replication factor (3) exceeds number >>>>>>>> of endpoints (1)" >>>>>>> >>>>>>> You already answered >>>>>>> "It always keeps showing only one node and mentions that it is handling >>>>>>> 100% of the load." >>>>>> >>>>>> Cluster think only one node is present in ring, it doesn't agree RF=3 >>>>>> it is >>>>>> expecting RF=1. >>>>>> Original Q: I m not exactly sure what is the problem. But >>>>>> Does nodetool ring show all the host? >>>>>> What is your seed list? >>>>>> Is bootstrapped node has seed ip of its own? >>>>>> AFAIK gossip work even without actively joining a ring. >>>>>> >>>>>>>> >>>>>>>> On Tue, Aug 2, 2011 at 7:21 AM, Aishwarya Venkataraman >>>>>>>> <cyberai...@gmail.com> wrote: >>>>>>>>> Replies inline. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Aishwarya >>>>>>>>> >>>>>>>>> On Tue, Aug 2, 2011 at 7:12 AM, Sorin Julean <sorin.jul...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Until someone answers with more details, few questions: >>>>>>>>>> 1. did you moved the system keyspace as well ? >>>>>>>>> Yes. But I deleted the LocationInfo* files under the system folder. >>>>>>>>> Shall I go ahead and delete the entire system folder ? >>>>>>>>> >>>>>>>>>> 2. the gossip IP of the new nodes are the same as the old ones ? >>>>>>>>> No. The Ip is different. >>>>>>>>> >>>>>>>>>> 3. which cassandra version are you running ? >>>>>>>>> I am using 0.8.1 >>>>>>>>> >>>>>>>>>> >>>>>>>>>> If 1. is yes and 2. is no, for a quick fix: take down the cluster, >>>>>>>>>> remove >>>>>>>>>> system keyspace, bring the cluster up and bootstrap the nodes. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Kind regards, >>>>>>>>>> Sorin >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Aug 2, 2011 at 2:53 PM, Aishwarya Venkataraman >>>>>>>>>> <cyberai...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> I recently migrated 400 GB of data that was on a different cassandra >>>>>>>>>>> cluster (3 node with RF= 3) to a new cluster. I have a 3 node >>>>>>>>>>> cluster >>>>>>>>>>> with replication factor set to three. When I run nodetool ring, it >>>>>>>>>>> does not show me all the nodes in the cluster. It always keeps >>>>>>>>>>> showing >>>>>>>>>>> only one node and mentions that it is handling 100% of the load. But >>>>>>>>>>> when I look at the logs, the nodes are able to talk to each other >>>>>>>>>>> via >>>>>>>>>>> the gossip protocol. Why does this happen ? Can you tell me what I >>>>>>>>>>> am >>>>>>>>>>> doing wrong ? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Aishwarya >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>>> >>>> >>>> >>> >>> >>> >>> -- >>> Jonathan Ellis >>> Project Chair, Apache Cassandra >>> co-founder of DataStax, the source for professional Cassandra support >>> http://www.datastax.com >>> > >
