Re: bootstrap question

2010-10-19 Thread Jonathan Ellis
I think this code has had some changes since beta2. Here is what it looks like in trunk: if (DatabaseDescriptor.getNonSystemTables().size() > 0) { bootstrap(token); assert !isBootstrapMode; // bootstrap will block until finished

bootstrap question

2010-10-19 Thread Yang
from line 396 of StorageService.java from the 0.7.0-beta2 source, it looks that when I boot up a completely new node, if there is not any keyspace defined in its storage.yaml, it would not even participate in the ring? in other words, let's say the cassandra instance currently has 10 nodes, and h

Re: Bootstrap question

2010-07-22 Thread Gary Dusbabek
On Wed, Jul 21, 2010 at 14:14, Anthony Molinaro wrote: > Sure, looks like that's in 0.6.4, so I'll probably just rebuild my server > based on the 0.6 branch, unless you want me to test just the patch for > 1221?  Most likely won't get a chance to try until tomorrow, so let me > know. > Either way

Re: Bootstrap question

2010-07-21 Thread Anthony Molinaro
Sure, looks like that's in 0.6.4, so I'll probably just rebuild my server based on the 0.6 branch, unless you want me to test just the patch for 1221? Most likely won't get a chance to try until tomorrow, so let me know. Thanks, -Anthony On Wed, Jul 21, 2010 at 06:58:13AM -0500, Gary Dusbabek w

Re: Bootstrap question

2010-07-21 Thread Gary Dusbabek
Anthony, I think you're seeing the results of CASSANDRA-1221. Each node has two connections with its peers. One connection is used for gossip, the other for exchanging commands. What you see with 1221 is the command socket getting 'stuck' after a peer is convicted by gossip and then recovers.

Re: Bootstrap question

2010-07-20 Thread Anthony Molinaro
I see this in the old nodes DEBUG [WRITE-/10.220.198.15] 2010-07-20 21:15:50,366 OutboundTcpConnection.java (line 142) attempting to connect to /10.220.198.15 INFO [GMFD:1] 2010-07-20 21:15:50,391 Gossiper.java (line 586) Node /10.220.198.15 is now part of the cluster INFO [GMFD:1] 2010-07-20 21

Re: Bootstrap question

2010-07-19 Thread Jonathan Ellis
What gets logged on the old nodes at debug, when you try to add a single new machine after a full cluster restart? Removing Location would blow away the nodes' token information... It should be safe if you set the InitialToken to what it used to be on each machine before bringing it up after nuki

Re: Bootstrap question

2010-07-18 Thread Anthony Molinaro
Yeah, I tried all that already and it didn't seem to work, no new nodes will bootstrap, which makes me think there's some saved state somewhere, preventing a new node from bootstrapping. I think maybe the Location sstables? Is it safe to nuke those on all hosts and restart everything? (I just don

Re: Bootstrap question

2010-07-18 Thread shimi
If I have problems with never ending bootstraping I do the following. I try each one if it doesn't help I try the next. It might not be the right thing to do but it worked for me. 1. Restart the bootstraping node 2. If I see streaming 0/ I restart the node and all the streaming nodes 3. Restar

Re: Bootstrap question

2010-07-17 Thread Anthony Molinaro
So still waiting for any sort of answer on this one. The cluster still refuses to do anything when I bring up new nodes. I shut down all the new nodes and am waiting. I'm guessing that maybe the old nodes have some state which needs to get cleared out? Is there anything I can do at this point?

Re: Bootstrap question

2010-07-16 Thread Anthony Molinaro
On Thu, Jul 15, 2010 at 10:45:08PM -0700, Anthony Molinaro wrote: > Is there something else I should try? The only thing I can think of > is deleting the system directory on the new node, and restarting, so > I'll try that and see if it does anything. So I tried this, it didn't do anything. The

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
Okay, so things were pretty messed up. I shut down all the new nodes, then the old nodes started doing the half the ring is down garbage which pretty much requires a full restart of everything. So I had to shut everything down, then bring the seed back, then the rest of the nodes, so they finally

Re: Bootstrap question

2010-07-15 Thread Jonathan Ellis
On Thu, Jul 15, 2010 at 3:28 PM, Anthony Molinaro wrote: > Is the fact that 2 new nodes are in the range messing it up? Probably. >  And if so > how do I recover (I'm thinking, shutdown new nodes 2,3,4,5, the bringing > up nodes 2,4, waiting for them to finish, then bringing up 3,5?). Yes. You

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
Oh, and looking at the load on the new machines it appears that New 2 and New 6 have gotten some data (although neither is in the ring yet). Not sure if that clears anything up though. -Anthony On Thu, Jul 15, 2010 at 01:28:06PM -0700, Anthony Molinaro wrote: > This is a cluster which is horri

Re: Bootstrap question

2010-07-15 Thread Anthony Molinaro
This is a cluster which is horribly imbalanced because I didn't assign initial tokens, so I'm adding 6 nodes with tokens according to the operations page (ie, i * (2^127/N) with N = 6). So here's what the ring will look like when bootstrap finishes 151901684708361811491018697

Re: Bootstrap question

2010-07-14 Thread Jonathan Ellis
Each node logs what token it is going to bootstrap to. Who owns the ranges that contain those tokens? On Wed, Jul 14, 2010 at 5:58 PM, Anthony Molinaro wrote: > Hi, > >  I have a 0.6.3 cluster which contains 6 nodes.  I added 6 new nodes > by setting AutoBootstrap to true and setting an InitialT

Bootstrap question

2010-07-14 Thread Anthony Molinaro
Hi, I have a 0.6.3 cluster which contains 6 nodes. I added 6 new nodes by setting AutoBootstrap to true and setting an InitialToken on each new node, then waiting for the "Bootstrapping" message in the log before starting another. Then I've been watching the logs on the old boxes waiting to se