Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-29 Thread John Watson
Opened a ticket: https://issues.apache.org/jira/browse/CASSANDRA-5525 On Mon, Apr 29, 2013 at 2:24 AM, aaron morton wrote: > is this understanding correct "we had a 12 node cluster with 256 vnodes on > each node (upgraded from 1.1), we added two additional nodes that streamed > so much data (60

Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-29 Thread John Watson
They were all restarted a couple times after adding 'num_tokens: 256' to cassandra.yaml. Yes and nodetool ring became 'unusable' due to all the new tokens. On Mon, Apr 29, 2013 at 10:24 AM, Sam Overton wrote: > Did you update num_tokens on the existing hosts and restart them, before > you trie

Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-29 Thread Sam Overton
Did you update num_tokens on the existing hosts and restart them, before you tried bootstrapping in the new node? If the new node tried to stream all the data in the cluster then this would be consistent with you having missed that step. You should see "Calculating new tokens" in the logs of the e

Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-29 Thread aaron morton
is this understanding correct "we had a 12 node cluster with 256 vnodes on each node (upgraded from 1.1), we added two additional nodes that streamed so much data (600+Gb when other nodes had 150-200GB) during the joining phase that they filled their local disks and had to be killed" ? Can you

Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-28 Thread John Watson
On Sun, Apr 28, 2013 at 2:19 PM, aaron morton wrote: > We're going to try running a shuffle before adding a new node again... >> maybe that will help >> > I don't think hurt but I doubt it will help. > We had to bail on shuffle since we need to add capacity ASAP and not in 20 days. > >It

Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-28 Thread aaron morton
> We're going to try running a shuffle before adding a new node again... maybe > that will help I don't think hurt but I doubt it will help. >> It seems when new nodes join, they are streamed *all* sstables in the >> cluster. > How many nodes did you join, what was the num_tokens ? Did yo

Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-26 Thread Bryan Talbot
I believe that "nodetool rebuild" is used to add a new datacenter, not just a new host to an existing cluster. Is that what you ran to add the node? -Bryan On Fri, Apr 26, 2013 at 1:27 PM, John Watson wrote: > Small relief we're not the only ones that had this issue. > > We're going to try r

Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-26 Thread John Watson
Small relief we're not the only ones that had this issue. We're going to try running a shuffle before adding a new node again... maybe that will help - John On Fri, Apr 26, 2013 at 5:07 AM, Francisco Nogueira Calmon Sobral < fsob...@igcorp.com.br> wrote: > I am using the same version and obser

Re: Adding nodes in 1.2 with vnodes requires huge disks

2013-04-26 Thread Francisco Nogueira Calmon Sobral
I am using the same version and observed something similar. I've added a new node, but the instructions from Datastax did not work for me. Then I ran "nodetool rebuild" on the new node. After finished this command, it contained two times the load of the other nodes. Even when I ran "nodetool cl

Adding nodes in 1.2 with vnodes requires huge disks

2013-04-25 Thread John Watson
After finally upgrading to 1.2.3 from 1.1.9, enabling vnodes, and running upgradesstables, I figured it would be safe to start adding nodes to the cluster. Guess not? It seems when new nodes join, they are streamed *all* sstables in the cluster. https://dl.dropbox.com/s/bampemkvlfck2dt/Screen%20S