Thank you all for your advice and good info. The node has died a couple of times with out of memory errors. I've restarted each time but it starts re - running compaction and then dies again.
Is there a better way to do this? On Apr 18, 2014 6:06 PM, "Steven A Robenalt" <srobe...@stanford.edu> wrote: > That's what I'd be doing, but I wouldn't expect it to run for 3 days this > time. My guess is that whatever was going wrong with the bootstrap when you > had 3 nodes starting at once was interfering with the completion of the 1 > remaining node of those 3. A clean bootstrap of a single node should > complete eventually, and I would think it'll be a lot less than 3 days. Our > database is much smaller than yours at the moment, so I can't really guide > you on how long it should take, but I'd think that others on the list with > similar database sizes might be able to give you a better idea. > > Steve > > > > On Fri, Apr 18, 2014 at 1:43 PM, Phil Burress <philburress...@gmail.com>wrote: > >> First, I just stopped 2 of the nodes and left one running. But this >> morning, I stopped that third node, cleared out the data, restarted and let >> it rejoin again. It appears streaming is done (according to netstats), >> right now it appears to be running compaction and building secondary index >> (according to compactionstats). Just sit and wait I guess? >> >> >> On Fri, Apr 18, 2014 at 2:23 PM, Steven A Robenalt <srobe...@stanford.edu >> > wrote: >> >>> Looking back through this email chain, it looks like Phil said he wasn't >>> using vnodes. >>> >>> For the record, we are using vnodes since we brought up our first >>> cluster, and have not seen any issues with bootstrapping new nodes either >>> to replace existing nodes, or to grow/shrink the cluster. We did adhere to >>> the caveats that new nodes should not be seed nodes, and that we should >>> allow each node to join the cluster completely before making any other >>> changes. >>> >>> Phil, when you dropped to adding just the single node to your cluster, >>> did you start over with the newly added node (blowing away the database >>> created on the previous startup), or did you shut down the other 2 added >>> nodes and leave the remaining one in progress to continue? >>> >>> Steve >>> >>> >>> On Fri, Apr 18, 2014 at 10:40 AM, Robert Coli <rc...@eventbrite.com>wrote: >>> >>>> On Fri, Apr 18, 2014 at 5:05 AM, Phil Burress <philburress...@gmail.com >>>> > wrote: >>>> >>>>> nodetool netstats shows 84 files. They are all at 100%. Nothing >>>>> showing in Pending or Active for Read Repair Stats. >>>>> >>>>> I'm assuming this means it's done. But it still shows "JOINING". Is >>>>> there an undocumented step I'm missing here? This whole process seems >>>>> broken to me. >>>>> >>>> >>>> Lately it seems like a lot more people than usual are : >>>> >>>> 1) using vnodes >>>> 2) unable to bootstrap new nodes >>>> >>>> If I were you, I would likely file a JIRA detailing your negative >>>> experience with this core functionality. >>>> >>>> =Rob >>>> >>>> >>>> >>> >>> >>> >>> -- >>> Steve Robenalt >>> Software Architect >>> HighWire | Stanford University >>> 425 Broadway St, Redwood City, CA 94063 >>> >>> srobe...@stanford.edu >>> http://highwire.stanford.edu >>> >>> >>> >>> >>> >>> >> > > > -- > Steve Robenalt > Software Architect > HighWire | Stanford University > 425 Broadway St, Redwood City, CA 94063 > > srobe...@stanford.edu > http://highwire.stanford.edu > > > > > >
