Did you update num_tokens on the existing hosts and restart them, before you tried bootstrapping in the new node? If the new node tried to stream all the data in the cluster then this would be consistent with you having missed that step.
You should see "Calculating new tokens" in the logs of the existing hosts if you performed that step correctly, and "nodetool ring" should show that the existing hosts each have 256 tokens which are contiguous in the ring. If you missed this step then the new node will be taking 256 tokens in a ring with only N tokens (1 per existing host) and so will end up with 256/(256+N) of the data (almost all of it). On 28 April 2013 23:01, John Watson <j...@disqus.com> wrote: > On Sun, Apr 28, 2013 at 2:19 PM, aaron morton <aa...@thelastpickle.com>wrote: > >> We're going to try running a shuffle before adding a new node again... >>> maybe that will help >>> >> I don't think hurt but I doubt it will help. >> > > We had to bail on shuffle since we need to add capacity ASAP and not in 20 > days. > > >> >> It seems when new nodes join, they are streamed *all* sstables in the >>>> cluster. >>>> >>>> >>>> >>>> How many nodes did you join, what was the num_tokens ? >> Did you notice streaming from all nodes (in the logs) or are you saying >> this in response to the cluster load increasing ? >> >> > Was only adding 2 nodes at the time (planning to add a total of 12.) > Starting with a cluster of 12, but now 11 since 1 node entered some weird > state when one of the new nodes ran out disk space. > num_tokens is set to 256 on all nodes. > Yes, nearly all current nodes were streaming to the new ones (which was > great until disk space was an issue.) > >> The purple line machine, I just stopped the joining process because >>>> the main cluster was dropping mutation messages at this point on a few >>>> nodes (and it still had dozens of sstables to stream.) >>>> >>>> Which were the new nodes ? >> Can you show the output from nodetool status? >> >> > The new nodes are the purple and gray lines above all the others. > > nodetool status doesn't show joining nodes. I think I saw a bug already > filed for this but I can't seem to find it. > > >> >> Cheers >> >> ----------------- >> Aaron Morton >> Freelance Cassandra Consultant >> New Zealand >> >> @aaronmorton >> http://www.thelastpickle.com >> >> On 27/04/2013, at 9:35 AM, Bryan Talbot <btal...@aeriagames.com> wrote: >> >> I believe that "nodetool rebuild" is used to add a new datacenter, not >> just a new host to an existing cluster. Is that what you ran to add the >> node? >> >> -Bryan >> >> >> >> On Fri, Apr 26, 2013 at 1:27 PM, John Watson <j...@disqus.com> wrote: >> >>> Small relief we're not the only ones that had this issue. >>> >>> We're going to try running a shuffle before adding a new node again... >>> maybe that will help >>> >>> - John >>> >>> >>> On Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral < >>> fsob...@igcorp.com.br> wrote: >>> >>>> I am using the same version and observed something similar. >>>> >>>> I've added a new node, but the instructions from Datastax did not work >>>> for me. Then I ran "nodetool rebuild" on the new node. After finished this >>>> command, it contained two times the load of the other nodes. Even when I >>>> ran "nodetool cleanup" on the older nodes, the situation was the same. >>>> >>>> The problem only seemed to disappear when "nodetool repair" was applied >>>> to all nodes. >>>> >>>> Regards, >>>> Francisco Sobral. >>>> >>>> >>>> >>>> >>>> On Apr 25, 2013, at 4:57 PM, John Watson <j...@disqus.com> wrote: >>>> >>>> After finally upgrading to 1.2.3 from 1.1.9, enabling vnodes, and >>>> running upgradesstables, I figured it would be safe to start adding nodes >>>> to the cluster. Guess not? >>>> >>>> It seems when new nodes join, they are streamed *all* sstables in the >>>> cluster. >>>> >>>> >>>> https://dl.dropbox.com/s/bampemkvlfck2dt/Screen%20Shot%202013-04-25%20at%2012.35.24%20PM.png >>>> >>>> The gray the line machine ran out disk space and for some reason >>>> cascaded into errors in the cluster about 'no host id' when trying to store >>>> hints for it (even though it hadn't joined yet). >>>> The purple line machine, I just stopped the joining process because the >>>> main cluster was dropping mutation messages at this point on a few nodes >>>> (and it still had dozens of sstables to stream.) >>>> >>>> I followed this: >>>> http://www.datastax.com/docs/1.2/operations/add_replace_nodes >>>> >>>> Is there something missing in that documentation? >>>> >>>> Thanks, >>>> >>>> John >>>> >>>> >>>> >>> >> >> > -- Sam Overton Acunu | http://www.acunu.com | @acunu
