Adding nodes
Hi all, Cluster of 2 DC and 24 nodes DC1 (RF3) = 12 nodes, 16 tokens each DC2 (RF3) = 12 nodes, 16 tokens each Adding 12 more nodes to DC1: I installed Cassandra (version is the same across all nodes) but, after the first node added, I couldn't seem to add any further nodes. I check nodetool status and the newly added node is UJ. It remains this way all day and only 86Gb of data is added to the node over the entire day (probably not yet complete). This seems a little slow and, more than a little inconvenient to only be able to add one node at a time - or at least one node every 2 minutes. When the cluster was created, I timed each node from service start to status UJ (having a UUID) and it was around 120 seconds. Of course there was no data. Is it possible I have some setting not correctly tuned? Thanks Marc
Re: Adding nodes
What version are you using? When you run `nodetool netstats` on the joining node, what is the output? How much data is there per node (presumably more than 86G)? On Thu, Jul 7, 2022 at 7:49 AM Marc Hoppins wrote: > Hi all, > > Cluster of 2 DC and 24 nodes > > DC1 (RF3) = 12 nodes, 16 tokens each > DC2 (RF3) = 12 nodes, 16 tokens each > > Adding 12 more nodes to DC1: I installed Cassandra (version is the same > across all nodes) but, after the first node added, I couldn't seem to add > any further nodes. > > I check nodetool status and the newly added node is UJ. It remains this > way all day and only 86Gb of data is added to the node over the entire day > (probably not yet complete). This seems a little slow and, more than a > little inconvenient to only be able to add one node at a time - or at least > one node every 2 minutes. When the cluster was created, I timed each node > from service start to status UJ (having a UUID) and it was around 120 > seconds. Of course there was no data. > > Is it possible I have some setting not correctly tuned? > > Thanks > > Marc >
Re: Adding nodes
86Gb (that's gigabits, which is 10.75GB, gigabytes) took an entire day seems obviously too long. I would check the network bandwidth, disk IO and CPU usage and find out what is the bottleneck. On 07/07/2022 15:48, Marc Hoppins wrote: Hi all, Cluster of 2 DC and 24 nodes DC1 (RF3) = 12 nodes, 16 tokens each DC2 (RF3) = 12 nodes, 16 tokens each Adding 12 more nodes to DC1: I installed Cassandra (version is the same across all nodes) but, after the first node added, I couldn't seem to add any further nodes. I check nodetool status and the newly added node is UJ. It remains this way all day and only 86Gb of data is added to the node over the entire day (probably not yet complete). This seems a little slow and, more than a little inconvenient to only be able to add one node at a time - or at least one node every 2 minutes. When the cluster was created, I timed each node from service start to status UJ (having a UUID) and it was around 120 seconds. Of course there was no data. Is it possible I have some setting not correctly tuned? Thanks Marc
RE: Adding nodes
Thank you for pointing that out. 85 gigabytes/gibibytes/GIGABYTES/GIBIBYTES/whatever name you care to give it CPU and bandwidth are not the problem. Version 4.0.3 but, as I stated, all nodes use the same version so the version is not important either. Existing nodes have 350-400+(choose whatever you want to call a gigabyte) The problem appears to be that adding new nodes is a serial process, which is fine when there is no data and each node is added within 2minutes. It is hardly practical in production. -Original Message- From: Bowen Song via user Sent: Thursday, July 7, 2022 8:43 PM To: user@cassandra.apache.org Subject: Re: Adding nodes EXTERNAL 86Gb (that's gigabits, which is 10.75GB, gigabytes) took an entire day seems obviously too long. I would check the network bandwidth, disk IO and CPU usage and find out what is the bottleneck. On 07/07/2022 15:48, Marc Hoppins wrote: > Hi all, > > Cluster of 2 DC and 24 nodes > > DC1 (RF3) = 12 nodes, 16 tokens each > DC2 (RF3) = 12 nodes, 16 tokens each > > Adding 12 more nodes to DC1: I installed Cassandra (version is the same > across all nodes) but, after the first node added, I couldn't seem to add any > further nodes. > > I check nodetool status and the newly added node is UJ. It remains this way > all day and only 86Gb of data is added to the node over the entire day > (probably not yet complete). This seems a little slow and, more than a > little inconvenient to only be able to add one node at a time - or at least > one node every 2 minutes. When the cluster was created, I timed each node > from service start to status UJ (having a UUID) and it was around 120 > seconds. Of course there was no data. > > Is it possible I have some setting not correctly tuned? > > Thanks > > Marc
