Nice catch! I’ve totally overlooked it. Thanks a lot! Stefano
On Sun, 15 Oct 2017 at 22:14, Jeff Jirsa <jji...@gmail.com> wrote: > (Should still be able to complete, unless you’re running out of disk or > memory or similar, but that’s why it’s streaming more than you expect) > > > -- > Jeff Jirsa > > > On Oct 15, 2017, at 1:51 PM, Jeff Jirsa <jji...@gmail.com> wrote: > > I > You’re adding the new node as rac3 > > The rack aware policy is going to make sure you get the rack diversity you > asked for by making sure one replica of each partition is in rac3, which is > going to blow up that instance > > > > -- > Jeff Jirsa > > > On Oct 15, 2017, at 1:42 PM, Stefano Ortolani <ostef...@gmail.com> wrote: > > Hi Jeff, > > this my third attempt bootstrapping the node so I tried several tricks > that might partially explain the output I am posting. > > * To make the bootstrap incremental, I have been throttling the streams on > all nodes to 1Mbits. I have selectively unthrottling one node at a time > hoping that would unlock some routines compacting away redundant data > (you'll see that nodetool netstats reports back fewer nodes than nodetool > status). > * Since compactions have had the tendency of getting stuck (hundreds > pending but none executing) in previous bootstraps, I've tried issuing a > manual "nodetool compact" on the boostrapping node. > > Having said that, this is the output of the commands, > > Thanks a lot, > Stefano > > *nodetool status* > Datacenter: DC1 > =============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID > Rack > UN X.Y.33.8 342.4 GB 256 ? > afaae414-30cc-439d-9785-1b7d35f74529 RAC1 > UN X.Y.81.4 325.98 GB 256 ? > 00a96a5d-3bfd-497f-91f3-973b75146162 RAC2 > UN X.Y.33.4 348.81 GB 256 ? > 1d8e6588-e25b-456a-8f29-0dedc35bda8e RAC1 > UN X.Y.33.5 384.99 GB 256 ? > 13d03fd2-7528-466b-b4b5-1b46508e2465 RAC1 > UN X.Y.81.5 336.27 GB 256 ? > aa161400-6c0e-4bde-bcb3-b2e7e7840196 RAC2 > UN X.Y.33.6 377.22 GB 256 ? > 43a393ba-6805-4e33-866f-124360174b28 RAC1 > UN X.Y.81.6 329.61 GB 256 ? > 4c3c64ae-ef4f-4986-9341-573830416997 RAC2 > UN X.Y.33.7 344.25 GB 256 ? > 03d81879-dc0d-4118-92e3-b3013dfde480 RAC1 > UN X.Y.81.7 324.93 GB 256 ? > 24bbf4b6-9427-4ed1-a751-a55cc24cc756 RAC2 > UN X.Y.81.1 323.8 GB 256 ? > 26244100-0565-4567-ae9c-0fc5346f5558 RAC2 > UJ X.Y.177.2 724.5 GB 256 ? > e269a06b-c0c0-43a6-922c-f04c98898e0d RAC3 > UN X.Y.81.2 337.83 GB 256 ? > 09e29429-15ff-44d6-9742-ac95c83c4d9e RAC2 > UN X.Y.81.3 326.4 GB 256 ? > feaa7b27-7ab8-4fc2-b64a-c9df3dd86d97 RAC2 > UN X.Y.33.3 350.4 GB 256 ? > cc115991-b7e7-4d06-87b5-8ad5efd45da5 RAC1 > > > *nodetool netstats -H | grep "Already received" -B 1* > /X.Y.81.4 > Receiving 1992 files, 103.68 GB total. Already received 515 files, > 23.32 GB total > -- > /X.Y.81.7 > Receiving 1936 files, 89.35 GB total. Already received 554 files, > 23.32 GB total > -- > /X.Y.81.5 > Receiving 1926 files, 95.69 GB total. Already received 545 files, > 23.31 GB total > -- > /X.Y.81.2 > Receiving 1992 files, 100.81 GB total. Already received 537 files, > 23.32 GB total > -- > /X.Y.81.3 > Receiving 1958 files, 104.72 GB total. Already received 503 files, > 23.31 GB total > -- > /X.Y.81.1 > Receiving 2034 files, 104.51 GB total. Already received 520 files, > 23.33 GB total > -- > /X.Y.81.6 > Receiving 1962 files, 96.19 GB total. Already received 547 files, > 23.32 GB total > -- > /X.Y.33.5 > Receiving 2121 files, 97.44 GB total. Already received 601 files, > 23.32 GB total > > *nodetool tpstats* > Pool Name Active Pending Completed Blocked > All time blocked > MutationStage 0 0 828367015 0 > 0 > ViewMutationStage 0 0 0 0 > 0 > ReadStage 0 0 0 0 > 0 > RequestResponseStage 0 0 13 0 > 0 > ReadRepairStage 0 0 0 0 > 0 > CounterMutationStage 0 0 0 0 > 0 > MiscStage 0 0 0 0 > 0 > CompactionExecutor 1 1 12150 0 > 0 > MemtableReclaimMemory 0 0 7368 0 > 0 > PendingRangeCalculator 0 0 14 0 > 0 > GossipStage 0 0 599329 0 > 0 > SecondaryIndexManagement 0 0 0 0 > 0 > HintsDispatcher 0 0 0 0 > 0 > MigrationStage 0 0 27 0 > 0 > MemtablePostFlush 0 0 8112 0 > 0 > ValidationExecutor 0 0 0 0 > 0 > Sampler 0 0 0 0 > 0 > MemtableFlushWriter 0 0 7368 0 > 0 > InternalResponseStage 0 0 25 0 > 0 > AntiEntropyStage 0 0 0 0 > 0 > CacheCleanupExecutor 0 0 0 0 > 0 > > Message type Dropped > READ 0 > RANGE_SLICE 0 > _TRACE 0 > HINT 0 > MUTATION 1 > COUNTER_MUTATION 0 > BATCH_STORE 0 > BATCH_REMOVE 0 > REQUEST_RESPONSE 0 > PAGED_RANGE 0 > READ_REPAIR 0 > > *nodetool compactionstats -H* > pending tasks: 776 > id compaction type keyspace > table completed total unit progress > 24d039f2-b1e6-11e7-ac57-3d25e38b2f5c Compaction keyspace_1 > table_1 4.85 GB 7.67 GB bytes 63.25% > Active compaction remaining time : n/a > > > On Sun, Oct 15, 2017 at 9:27 PM, Jeff Jirsa <jji...@gmail.com> wrote: > >> Can you post (anonymize as needed) nodetool status, nodetool netstats, >> nodetool tpstats, and nodetool compctionstats ? >> >> -- >> Jeff Jirsa >> >> >> On Oct 15, 2017, at 1:14 PM, Stefano Ortolani <ostef...@gmail.com> wrote: >> >> Hi Jeff, >> >> that would be 3.0.15, single disk, vnodes enabled (num_tokens 256). >> >> Stefano >> >> On Sun, Oct 15, 2017 at 9:11 PM, Jeff Jirsa <jji...@gmail.com> wrote: >> >>> What version? >>> >>> Single disk or JBOD? >>> >>> Vnodes? >>> >>> -- >>> Jeff Jirsa >>> >>> >>> On Oct 15, 2017, at 12:49 PM, Stefano Ortolani <ostef...@gmail.com> >>> wrote: >>> >>> Hi all, >>> >>> I have been trying "-Dcassandra.disable_stcs_in_l0=true", but no luck >>> so far. >>> Based on the source code it seems that this option doesn't affect >>> compactions while bootstrapping. >>> >>> I am getting quite confused as it seems I am not able to bootstrap a >>> node if I don't have at least 6/7 times the disk space used by other nodes. >>> This is weird. The host I am bootstrapping is using a SSD. Also >>> compaction throughput is unthrottled (set to 0) and the compacting threads >>> are set to 8. >>> Nevertheless, primary ranges from other nodes are being streamed, but >>> data is never compacted away. >>> >>> Does anybody know anything else I could try? >>> >>> Cheers, >>> Stefano >>> >>> On Fri, Oct 13, 2017 at 3:58 PM, Stefano Ortolani <ostef...@gmail.com> >>> wrote: >>> >>>> Other little update: at the same time I see the number of pending tasks >>>> stuck (in this case at 1847); restarting the node doesn't help, so I can't >>>> really force the node to "digest" all those compactions. In the meanwhile >>>> the disk occupied is already twice the average load I have on other nodes. >>>> >>>> Feeling more and more puzzled here :S >>>> >>>> On Fri, Oct 13, 2017 at 1:28 PM, Stefano Ortolani <ostef...@gmail.com> >>>> wrote: >>>> >>>>> I have been trying to add another node to the cluster (after upgrading >>>>> to 3.0.15) and I just noticed through "nodetool netstats" that all nodes >>>>> have been streaming to the joining node approx 1/3 of their SSTables, >>>>> basically their whole primary range (using RF=3)? >>>>> >>>>> Is this expected/normal? >>>>> I was under the impression only the necessary SSTables were going to >>>>> be streamed... >>>>> >>>>> Thanks for the help, >>>>> Stefano >>>>> >>>>> >>>>> On Wed, Aug 23, 2017 at 1:37 PM, kurt greaves <k...@instaclustr.com> >>>>> wrote: >>>>> >>>>>> But if it also streams, it means I'd still be under-pressure if I am >>>>>>> not mistaken. I am under the assumption that the compactions are the >>>>>>> by-product of streaming too many SStables at the same time, and not >>>>>>> because >>>>>>> of my current write load. >>>>>>> >>>>>> Ah yeah I wasn't thinking about the capacity problem, more of the >>>>>> performance impact from the node being backed up with compactions. If you >>>>>> haven't already, you should try disable stcs in l0 on the joining node. >>>>>> You >>>>>> will likely still need to do a lot of compactions, but generally they >>>>>> should be smaller. The option is -Dcassandra.disable_stcs_in_l0=true >>>>>> >>>>>>> I just noticed you were mentioning L1 tables too. Why would that >>>>>>> affect the disk footprint? >>>>>> >>>>>> If you've been doing a lot of STCS in L0, you generally end up with >>>>>> some large SSTables. These will eventually have to be compacted with L1. >>>>>> Could also be suffering the problem of streamed SSTables causing large >>>>>> cross-level compactions in the higher levels as well. >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> >
