> We're going to try running a shuffle before adding a new node again... maybe > that will help I don't think hurt but I doubt it will help.
>> It seems when new nodes join, they are streamed *all* sstables in the >> cluster. > How many nodes did you join, what was the num_tokens ? Did you notice streaming from all nodes (in the logs) or are you saying this in response to the cluster load increasing ? >> The purple line machine, I just stopped the joining process because the main >> cluster was dropping mutation messages at this point on a few nodes (and it >> still had dozens of sstables to stream.) Which were the new nodes ? Can you show the output from nodetool status? Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 27/04/2013, at 9:35 AM, Bryan Talbot <btal...@aeriagames.com> wrote: > I believe that "nodetool rebuild" is used to add a new datacenter, not just a > new host to an existing cluster. Is that what you ran to add the node? > > -Bryan > > > > On Fri, Apr 26, 2013 at 1:27 PM, John Watson <j...@disqus.com> wrote: > Small relief we're not the only ones that had this issue. > > We're going to try running a shuffle before adding a new node again... maybe > that will help > > - John > > > On Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral > <fsob...@igcorp.com.br> wrote: > I am using the same version and observed something similar. > > I've added a new node, but the instructions from Datastax did not work for > me. Then I ran "nodetool rebuild" on the new node. After finished this > command, it contained two times the load of the other nodes. Even when I ran > "nodetool cleanup" on the older nodes, the situation was the same. > > The problem only seemed to disappear when "nodetool repair" was applied to > all nodes. > > Regards, > Francisco Sobral. > > > > > On Apr 25, 2013, at 4:57 PM, John Watson <j...@disqus.com> wrote: > >> After finally upgrading to 1.2.3 from 1.1.9, enabling vnodes, and running >> upgradesstables, I figured it would be safe to start adding nodes to the >> cluster. Guess not? >> >> It seems when new nodes join, they are streamed *all* sstables in the >> cluster. >> >> https://dl.dropbox.com/s/bampemkvlfck2dt/Screen%20Shot%202013-04-25%20at%2012.35.24%20PM.png >> >> The gray the line machine ran out disk space and for some reason cascaded >> into errors in the cluster about 'no host id' when trying to store hints for >> it (even though it hadn't joined yet). >> The purple line machine, I just stopped the joining process because the main >> cluster was dropping mutation messages at this point on a few nodes (and it >> still had dozens of sstables to stream.) >> >> I followed this: >> http://www.datastax.com/docs/1.2/operations/add_replace_nodes >> >> Is there something missing in that documentation? >> >> Thanks, >> >> John > > >
