nodetool status: Datacenter: datacenter1 ======================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.10.20.27 1.89 TB 256 25.4% 76023cdd-c42d-4068-8b53-ae94584b8b04 rack1 UN 10.10.20.62 1.83 TB 256 25.5% 84b47313-da75-4519-94f3-3951d554a3e5 rack1 UN 10.10.20.47 1.87 TB 256 24.7% bcd51a92-3150-41ae-9c51-104ea154f6fa rack1 UN 10.10.20.45 1.7 TB 256 22.6% 8d6bce33-8179-4660-8443-2cf822074ca4 rack1 UN 10.10.20.15 1.86 TB 256 24.5% 01a01f07-4df2-4c87-98e9-8dd38b3e4aee rack1 UN 10.10.20.31 1.87 TB 256 24.9% 1435acf9-c64d-4bcd-b6a4-abcec209815e rack1 UN 10.10.20.35 1.86 TB 256 25.8% 17cb8772-2444-46ff-8525-33746514727d rack1 UN 10.10.20.51 1.89 TB 256 25.0% 0343cd58-3686-465f-8280-56fb72d161e2 rack1 UN 10.10.20.19 1.91 TB 256 25.5% 30ddf003-4d59-4a3e-85fa-e94e4adba1cb rack1 UN 10.10.20.39 1.93 TB 256 26.0% b7d44c26-4d75-4d36-a779-b7e7bdaecbc9 rack1 UN 10.10.20.52 1.81 TB 256 25.4% 6b5aca07-1b14-4bc2-a7ba-96f026fa0e4e rack1 UN 10.10.20.22 1.89 TB 256 24.8% 46af9664-8975-4c91-847f-3f7b8f8d5ce2 rack1
Note: The new node is not part of the above list. nodetool compactionstats: pending tasks: 1649 compaction type keyspace column family completed total unit progress Compaction iprod customerorder 1682804084 17956558077 bytes 9.37% Compaction prodgatecustomerorder 1664239271 1693502275 bytes 98.27% Compaction qa_config_bkupfixsessionconfig_hist 2443 27253 bytes 8.96% Compaction prodgatecustomerorder_hist 1770577280 5026699390 bytes 35.22% Compaction iprodgatecustomerorder_hist 2959560205 312350192622 bytes 0.95% On Tue, Aug 5, 2014 at 11:37 AM, Mark Reddy <mark.re...@boxever.com> wrote: > Yes num_tokens is set to 256. initial_token is blank on all nodes >> including the new one. > > > Ok so you have num_tokens set to 256 for all nodes with initial_token > commented out, this means you are using vnodes and the new node will > automatically grab a list of tokens to take over responsibility for. > > Pool Name Active Pending Completed Blocked >> All time blocked >> FlushWriter 0 0 1136 0 >> 512 >> >> Looks like about 50% of flushes are blocked. >> > > This is a problem as it indicates that the IO system cannot keep up. > > Just ran this on the new node: >> nodetool netstats | grep "Streaming from" | wc -l >> 10 > > > This is normal as the new node will most likely take tokens from all nodes > in the cluster. > > Sorry for the multiple updates, but another thing I found was all the >> other existing nodes have themselves in the seeds list, but the new node >> does not have itself in the seeds list. Can that cause this issue? > > > Seeds are only used when a new node is bootstrapping into the cluster and > needs a set of ips to contact and discover the cluster, so this would have > no impact on data sizes or streaming. In general it would be considered > best practice to have a set of 2-3 seeds from each data center, with all > nodes having the same seed list. > > > What is the current output of 'nodetool compactionstats'? Could you also > paste the output of nodetool status <keyspace>? > > Mark > > > > On Tue, Aug 5, 2014 at 3:59 PM, Ruchir Jha <ruchir....@gmail.com> wrote: > >> Sorry for the multiple updates, but another thing I found was all the >> other existing nodes have themselves in the seeds list, but the new node >> does not have itself in the seeds list. Can that cause this issue? >> >> >> On Tue, Aug 5, 2014 at 10:30 AM, Ruchir Jha <ruchir....@gmail.com> wrote: >> >>> Just ran this on the new node: >>> >>> nodetool netstats | grep "Streaming from" | wc -l >>> 10 >>> >>> Seems like the new node is receiving data from 10 other nodes. Is that >>> expected in a vnodes enabled environment? >>> >>> Ruchir. >>> >>> >>> >>> On Tue, Aug 5, 2014 at 10:21 AM, Ruchir Jha <ruchir....@gmail.com> >>> wrote: >>> >>>> Also not sure if this is relevant but just noticed the nodetool tpstats >>>> output: >>>> >>>> Pool Name Active Pending Completed Blocked >>>> All time blocked >>>> FlushWriter 0 0 1136 0 >>>> 512 >>>> >>>> Looks like about 50% of flushes are blocked. >>>> >>>> >>>> On Tue, Aug 5, 2014 at 10:14 AM, Ruchir Jha <ruchir....@gmail.com> >>>> wrote: >>>> >>>>> Yes num_tokens is set to 256. initial_token is blank on all nodes >>>>> including the new one. >>>>> >>>>> >>>>> On Tue, Aug 5, 2014 at 10:03 AM, Mark Reddy <mark.re...@boxever.com> >>>>> wrote: >>>>> >>>>>> My understanding was that if initial_token is left empty on the new >>>>>>> node, it just contacts the heaviest node and bisects its token range. >>>>>> >>>>>> >>>>>> If you are using vnodes and you have num_tokens set to 256 the new >>>>>> node will take token ranges dynamically. What is the configuration of >>>>>> your >>>>>> other nodes, are you setting num_tokens or initial_token on those? >>>>>> >>>>>> >>>>>> Mark >>>>>> >>>>>> >>>>>> On Tue, Aug 5, 2014 at 2:57 PM, Ruchir Jha <ruchir....@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Thanks Patricia for your response! >>>>>>> >>>>>>> On the new node, I just see a lot of the following: >>>>>>> >>>>>>> INFO [FlushWriter:75] 2014-08-05 09:53:04,394 Memtable.java (line >>>>>>> 400) Writing Memtable >>>>>>> INFO [CompactionExecutor:3] 2014-08-05 09:53:11,132 >>>>>>> CompactionTask.java (line 262) Compacted 12 sstables to >>>>>>> >>>>>>> so basically it is just busy flushing, and compacting. Would you >>>>>>> have any ideas on why the 2x disk space blow up. My understanding was >>>>>>> that >>>>>>> if initial_token is left empty on the new node, it just contacts the >>>>>>> heaviest node and bisects its token range. And the heaviest node is >>>>>>> around >>>>>>> 2.1 TB, and the new node is already at 4 TB. Could this be because >>>>>>> compaction is falling behind? >>>>>>> >>>>>>> Ruchir >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 4, 2014 at 7:23 PM, Patricia Gorla < >>>>>>> patri...@thelastpickle.com> wrote: >>>>>>> >>>>>>>> Ruchir, >>>>>>>> >>>>>>>> What exactly are you seeing in the logs? Are you running major >>>>>>>> compactions on the new bootstrapping node? >>>>>>>> >>>>>>>> With respect to the seed list, it is generally advisable to use 3 >>>>>>>> seed nodes per AZ / DC. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Aug 4, 2014 at 11:41 AM, Ruchir Jha <ruchir....@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I am trying to bootstrap the thirteenth node in a 12 node cluster >>>>>>>>> where the average data size per node is about 2.1 TB. The bootstrap >>>>>>>>> streaming has been going on for 2 days now, and the disk size on the >>>>>>>>> new >>>>>>>>> node is already above 4 TB and still going. Is this because the new >>>>>>>>> node is >>>>>>>>> running major compactions while the streaming is going on? >>>>>>>>> >>>>>>>>> One thing that I noticed that seemed off was the seeds property in >>>>>>>>> the yaml of the 13th node comprises of 1..12. Where as the seeds >>>>>>>>> property >>>>>>>>> on the existing 12 nodes consists of all the other nodes except the >>>>>>>>> thirteenth node. Is this an issue? >>>>>>>>> >>>>>>>>> Any other insight is appreciated? >>>>>>>>> >>>>>>>>> Ruchir. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Patricia Gorla >>>>>>>> @patriciagorla >>>>>>>> >>>>>>>> Consultant >>>>>>>> Apache Cassandra Consulting >>>>>>>> http://www.thelastpickle.com <http://thelastpickle.com> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >