Yeah, I tried all that already and it didn't seem to work, no new nodes will bootstrap, which makes me think there's some saved state somewhere, preventing a new node from bootstrapping. I think maybe the Location sstables? Is it safe to nuke those on all hosts and restart everything? (I just don't want to lose actual data).
Thanks for the ideas, -Anthony On Sun, Jul 18, 2010 at 08:09:45PM +0300, shimi wrote: > If I have problems with never ending bootstraping I do the following. I try > each one if it doesn't help I try the next. It might not be the right thing > to do but it worked for me. > > 1. Restart the bootstraping node > 2. If I see streaming 0/xxxx I restart the node and all the streaming nodes > 3. Restart all the nodes > 4. If there is data in the bootstraing node I delete it before I restart. > > Good luck > Shimi > > On Sun, Jul 18, 2010 at 12:21 AM, Anthony Molinaro < > antho...@alumni.caltech.edu> wrote: > > > So still waiting for any sort of answer on this one. The cluster still > > refuses to do anything when I bring up new nodes. I shut down all the > > new nodes and am waiting. I'm guessing that maybe the old nodes have > > some state which needs to get cleared out? Is there anything I can do > > at this point? Are there alternate strategies for bootstrapping I can > > try? (For instance can I just scp all the sstables to all the new > > nodes and do a repair, would that actually work?). > > > > Anyone seen this sort of issue? All this is with 0.6.3 so I assume > > eventually others will see this issue. > > > > -Anthony > > > > On Thu, Jul 15, 2010 at 10:45:08PM -0700, Anthony Molinaro wrote: > > > Okay, so things were pretty messed up. I shut down all the new nodes, > > > then the old nodes started doing the half the ring is down garbage which > > > pretty much requires a full restart of everything. So I had to shut > > > everything down, then bring the seed back, then the rest of the nodes, > > > so they finally all agreed on the ring again. > > > > > > Then I started one of the new nodes, and have been watching the logs, so > > > far 2 hours since the "Bootstrapping" message appeared in the new > > > log and nothing has happened. No anticompaction messages anywhere, > > there's > > > one node compacting, but its on the other end of the ring, so no where > > near > > > that new node. I'm wondering if it will ever get data at this point. > > > > > > Is there something else I should try? The only thing I can think of > > > is deleting the system directory on the new node, and restarting, so > > > I'll try that and see if it does anything. > > > > > > -Anthony > > > > > > On Thu, Jul 15, 2010 at 03:43:49PM -0500, Jonathan Ellis wrote: > > > > On Thu, Jul 15, 2010 at 3:28 PM, Anthony Molinaro > > > > <antho...@alumni.caltech.edu> wrote: > > > > > Is the fact that 2 new nodes are in the range messing it up? > > > > > > > > Probably. > > > > > > > > > And if so > > > > > how do I recover (I'm thinking, shutdown new nodes 2,3,4,5, the > > bringing > > > > > up nodes 2,4, waiting for them to finish, then bringing up 3,5?). > > > > > > > > Yes. > > > > > > > > You might have to restart the old nodes too to clear out the confusion. > > > > > > > > -- > > > > Jonathan Ellis > > > > Project Chair, Apache Cassandra > > > > co-founder of Riptano, the source for professional Cassandra support > > > > http://riptano.com > > > > > > -- > > > ------------------------------------------------------------------------ > > > Anthony Molinaro <antho...@alumni.caltech.edu> > > > > -- > > ------------------------------------------------------------------------ > > Anthony Molinaro <antho...@alumni.caltech.edu> > > -- ------------------------------------------------------------------------ Anthony Molinaro <antho...@alumni.caltech.edu>
