In storage-conf I see this comment [1] from which I understand that the recommended way to bootstrap a new node is to set AutoBootstrap=true and remove itself from the seeds list. Moreover, I did try to set AutoBootstrap=true and have the node in its own seeds list, but it would not bootstrap. I don't recall the exact message but it was something like "I found myself in the seeds list therefore I'm not going to bootstrap even though AutoBootstrap is true".
[1] <!-- ~ Turn on to make new [non-seed] nodes automatically migrate the right data ~ to themselves. (If no InitialToken is specified, they will pick one ~ such that they will get half the range of the most-loaded node.) ~ If a node starts up without bootstrapping, it will mark itself bootstrapped ~ so that you can't subsequently accidently bootstrap a node with ~ data on it. (You can reset this by wiping your data and commitlog ~ directories.) ~ ~ Off by default so that new clusters and upgraders from 0.4 don't ~ bootstrap immediately. You should turn this on when you start adding ~ new nodes to a cluster that already has data on it. (If you are upgrading ~ from 0.4, start your cluster with it off once before changing it to true. ~ Otherwise, no data will be lost but you will incur a lot of unnecessary ~ I/O before your cluster starts up.) --> <AutoBootstrap>false</AutoBootstrap> On Wed, Jan 5, 2011 at 4:58 PM, David Boxenhorn <da...@lookin2.com> wrote: > If "seed list should be the same across the cluster" that means that nodes > *should* have themselves as a seed. If that doesn't work for Ran, then that > is the first problem, no? > > > On Wed, Jan 5, 2011 at 3:56 PM, Jake Luciani <jak...@gmail.com> wrote: > >> Well your ring issues don't make sense to me, seed list should be the same >> across the cluster. >> I'm just thinking of other things to try, non-boostrapped nodes should >> join the ring instantly but reads will fail if you aren't using quorum. >> >> >> On Wed, Jan 5, 2011 at 8:51 AM, Ran Tavory <ran...@gmail.com> wrote: >> >>> I haven't tried repair. Should I? >>> On Jan 5, 2011 3:48 PM, "Jake Luciani" <jak...@gmail.com> wrote: >>> > Have you tried not bootstrapping but setting the token and manually >>> calling >>> > repair? >>> > >>> > On Wed, Jan 5, 2011 at 7:07 AM, Ran Tavory <ran...@gmail.com> wrote: >>> > >>> >> My conclusion is lame: I tried this on several hosts and saw the same >>> >> behavior, the only way I was able to join new nodes was to first start >>> them >>> >> when they are *not in* their own seeds list and after they >>> >> finish transferring the data, then restart them with themselves *in* >>> their >>> >> own seeds list. After doing that the node would join the ring. >>> >> This is either my misunderstanding or a bug, but the only place I >>> found it >>> >> documented stated that the new node should not be in its own seeds >>> list. >>> >> Version 0.6.6. >>> >> >>> >> On Wed, Jan 5, 2011 at 10:35 AM, David Boxenhorn <da...@lookin2.com >>> >wrote: >>> >> >>> >>> My nodes all have themselves in their list of seeds - always did - >>> and >>> >>> everything works. (You may ask why I did this. I don't know, I must >>> have >>> >>> copied it from an example somewhere.) >>> >>> >>> >>> On Wed, Jan 5, 2011 at 9:42 AM, Ran Tavory <ran...@gmail.com> wrote: >>> >>> >>> >>>> I was able to make the node join the ring but I'm confused. >>> >>>> What I did is, first when adding the node, this node was not in the >>> seeds >>> >>>> list of itself. AFAIK this is how it's supposed to be. So it was >>> able to >>> >>>> transfer all data to itself from other nodes but then it stayed in >>> the >>> >>>> bootstrapping state. >>> >>>> So what I did (and I don't know why it works), is add this node to >>> the >>> >>>> seeds list in its own storage-conf.xml file. Then restart the server >>> and >>> >>>> then I finally see it in the ring... >>> >>>> If I had added the node to the seeds list of itself when first >>> joining >>> >>>> it, it would not join the ring but if I do it in two phases it did >>> work. >>> >>>> So it's either my misunderstanding or a bug... >>> >>>> >>> >>>> >>> >>>> On Wed, Jan 5, 2011 at 7:14 AM, Ran Tavory <ran...@gmail.com> >>> wrote: >>> >>>> >>> >>>>> The new node does not see itself as part of the ring, it sees all >>> others >>> >>>>> but itself, so from that perspective the view is consistent. >>> >>>>> The only problem is that the node never finishes to bootstrap. It >>> stays >>> >>>>> in this state for hours (It's been 20 hours now...) >>> >>>>> >>> >>>>> >>> >>>>> $ bin/nodetool -p 9004 -h localhost streams >>> >>>>>> Mode: Bootstrapping >>> >>>>>> Not sending any streams. >>> >>>>>> Not receiving any streams. >>> >>>>> >>> >>>>> >>> >>>>> On Wed, Jan 5, 2011 at 1:20 AM, Nate McCall <n...@riptano.com> >>> wrote: >>> >>>>> >>> >>>>>> Does the new node have itself in the list of seeds per chance? >>> This >>> >>>>>> could cause some issues if so. >>> >>>>>> >>> >>>>>> On Tue, Jan 4, 2011 at 4:10 PM, Ran Tavory <ran...@gmail.com> >>> wrote: >>> >>>>>> > I'm still at lost. I haven't been able to resolve this. I tried >>> >>>>>> > adding another node at a different location on the ring but this >>> node >>> >>>>>> > too remains stuck in the bootstrapping state for many hours >>> without >>> >>>>>> > any of the other nodes being busy with anti compaction or >>> anything >>> >>>>>> > else. I don't know what's keeping it from finishing the >>> bootstrap,no >>> >>>>>> > CPU, no io, files were already streamed so what is it waiting >>> for? >>> >>>>>> > I read the release notes of 0.6.7 and 0.6.8 and there didn't >>> seem to >>> >>>>>> > be anything addressing a similar issue so I figured there was no >>> >>>>>> point >>> >>>>>> > in upgrading. But let me know if you think there is. >>> >>>>>> > Or any other advice... >>> >>>>>> > >>> >>>>>> > On Tuesday, January 4, 2011, Ran Tavory <ran...@gmail.com> >>> wrote: >>> >>>>>> >> Thanks Jake, but unfortunately the streams directory is empty >>> so I >>> >>>>>> don't think that any of the nodes is anti-compacting data right >>> now or had >>> >>>>>> been in the past 5 hours. It seems that all the data was already >>> transferred >>> >>>>>> to the joining host but the joining node, after having received >>> the data >>> >>>>>> would still remain in bootstrapping mode and not join the cluster. >>> I'm not >>> >>>>>> sure that *all* data was transferred (perhaps other nodes need to >>> transfer >>> >>>>>> more data) but nothing is actually happening so I assume all has >>> been moved. >>> >>>>>> >> Perhaps it's a configuration error from my part. Should I use I >>> use >>> >>>>>> AutoBootstrap=true ? Anything else I should look out for in the >>> >>>>>> configuration file or something else? >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> On Tue, Jan 4, 2011 at 4:08 PM, Jake Luciani <jak...@gmail.com >>> > >>> >>>>>> wrote: >>> >>>>>> >> >>> >>>>>> >> In 0.6, locate the node doing anti-compaction and look in the >>> >>>>>> "streams" subdirectory in the keyspace data dir to monitor the >>> >>>>>> anti-compaction progress (it puts new SSTables for bootstrapping >>> node in >>> >>>>>> there) >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> On Tue, Jan 4, 2011 at 8:01 AM, Ran Tavory <ran...@gmail.com> >>> >>>>>> wrote: >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> Running nodetool decommission didn't help. Actually the node >>> refused >>> >>>>>> to decommission itself (b/c it wasn't part of the ring). So I >>> simply stopped >>> >>>>>> the process, deleted all the data directories and started it >>> again. It >>> >>>>>> worked in the sense of the node bootstrapped again but as before, >>> after it >>> >>>>>> had finished moving the data nothing happened for a long time (I'm >>> still >>> >>>>>> waiting, but nothing seems to be happening). >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> Any hints how to analyze a "stuck" bootstrapping node??thanks >>> >>>>>> >> On Tue, Jan 4, 2011 at 1:51 PM, Ran Tavory <ran...@gmail.com> >>> >>>>>> wrote: >>> >>>>>> >> Thanks Shimi, so indeed anticompaction was run on one of the >>> other >>> >>>>>> nodes from the same DC but to my understanding it has already >>> ended. A few >>> >>>>>> hour ago... >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> I plenty of log messages such as [1] which ended a couple of >>> hours >>> >>>>>> ago, and I've seen the new node streaming and accepting the data >>> from the >>> >>>>>> node which performed the anticompaction and so far it was normal >>> so it >>> >>>>>> seemed that data is at its right place. But now the new node seems >>> sort of >>> >>>>>> stuck. None of the other nodes is anticompacting right now or had >>> been >>> >>>>>> anticompacting since then. >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> The new node's CPU is close to zero, it's iostats are almost >>> zero so >>> >>>>>> I can't find another bottleneck that would keep it hanging. >>> >>>>>> >> On the IRC someone suggested I'd maybe retry to join this node, >>> >>>>>> e.g. decommission and rejoin it again. I'll try it now... >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> [1] INFO [COMPACTION-POOL:1] 2011-01-04 04:04:09,721 >>> >>>>>> CompactionManager.java (line 338) AntiCompacting >>> >>>>>> >>> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvAds-6449-Data.db')] >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:18,683 >>> >>>>>> CompactionManager.java (line 338) AntiCompacting >>> >>>>>> >>> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3874-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3873-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvImpressions-3876-Data.db')] >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:19,132 >>> >>>>>> CompactionManager.java (line 338) AntiCompacting >>> >>>>>> >>> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-951-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-976-Data.db'),org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvRatings-978-Data.db')] >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> INFO [COMPACTION-POOL:1] 2011-01-04 04:34:26,486 >>> >>>>>> CompactionManager.java (line 338) AntiCompacting >>> >>>>>> >>> [org.apache.cassandra.io.SSTableReader(path='/outbrain/cassandra/data/outbrain_kvdb/KvAds-6449-Data.db')] >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> On Tue, Jan 4, 2011 at 12:45 PM, shimi <shim...@gmail.com> >>> wrote: >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> In my experience most of the time it takes for a node to join >>> the >>> >>>>>> cluster is the anticompaction on the other nodes. The streaming >>> part is very >>> >>>>>> fast. >>> >>>>>> >> Check the other nodes logs to see if there is any node doing >>> >>>>>> anticompaction.I don't remember how much data I had in the cluster >>> when I >>> >>>>>> needed to add/remove nodes. I do remember that it took a few >>> hours. >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> >> The node will join the ring only when it will finish the >>> bootstrap. >>> >>>>>> >> -- >>> >>>>>> >> /Ran >>> >>>>>> >> >>> >>>>>> >> >>> >>>>>> > >>> >>>>>> > -- >>> >>>>>> > /Ran >>> >>>>>> > >>> >>>>>> >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> /Ran >>> >>>>> >>> >>>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> /Ran >>> >>>> >>> >>>> >>> >>> >>> >> >>> >> >>> >> -- >>> >> /Ran >>> >> >>> >> >>> >> >> > -- /Ran
