Ifconfig shows RX of 1.1T. This doesn't seem to fit with the LOAD of 145GiB (nodetool status), unless I am reading that wrong...and the fact that this node still has a status of UJ.
Netstats on this node shows (other than : Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool Name Active Pending Completed Dropped Large messages n/a 0 0 0 Small messages n/a 53 569755545 15740262 Gossip messages n/a 0 288878 2 None of this addresses the issue of not being able to add more nodes. -----Original Message----- From: Bowen Song via user <user@cassandra.apache.org> Sent: Friday, July 8, 2022 11:47 AM To: user@cassandra.apache.org Subject: Re: Adding nodes EXTERNAL I would assume that's 85 GB (i.e. gigabytes) then. Which is approximately 79 GiB (i.e. gibibytes). This still sounds awfully slow - less than 1MB/s over a full day (24 hours). You said CPU and network aren't the bottleneck. Have you checked the disk IO? Also, be mindful with CPU usage. It can still be a bottleneck if one thread uses 100% of a CPU core while all other cores are idle. On 08/07/2022 07:09, Marc Hoppins wrote: > Thank you for pointing that out. > > 85 gigabytes/gibibytes/GIGABYTES/GIBIBYTES/whatever name you care to > give it > > CPU and bandwidth are not the problem. > > Version 4.0.3 but, as I stated, all nodes use the same version so the version > is not important either. > > Existing nodes have 350-400+(choose whatever you want to call a > gigabyte) > > The problem appears to be that adding new nodes is a serial process, which is > fine when there is no data and each node is added within 2minutes. It is > hardly practical in production. > > -----Original Message----- > From: Bowen Song via user <user@cassandra.apache.org> > Sent: Thursday, July 7, 2022 8:43 PM > To: user@cassandra.apache.org > Subject: Re: Adding nodes > > EXTERNAL > > > 86Gb (that's gigabits, which is 10.75GB, gigabytes) took an entire day seems > obviously too long. I would check the network bandwidth, disk IO and CPU > usage and find out what is the bottleneck. > > On 07/07/2022 15:48, Marc Hoppins wrote: >> Hi all, >> >> Cluster of 2 DC and 24 nodes >> >> DC1 (RF3) = 12 nodes, 16 tokens each >> DC2 (RF3) = 12 nodes, 16 tokens each >> >> Adding 12 more nodes to DC1: I installed Cassandra (version is the same >> across all nodes) but, after the first node added, I couldn't seem to add >> any further nodes. >> >> I check nodetool status and the newly added node is UJ. It remains this way >> all day and only 86Gb of data is added to the node over the entire day >> (probably not yet complete). This seems a little slow and, more than a >> little inconvenient to only be able to add one node at a time - or at least >> one node every 2 minutes. When the cluster was created, I timed each node >> from service start to status UJ (having a UUID) and it was around 120 >> seconds. Of course there was no data. >> >> Is it possible I have some setting not correctly tuned? >> >> Thanks >> >> Marc
