I'm assuming that your replication factor is 3. If that's the case, did you intentionally put this node in rack 4? Typically, you want to add nodes in multiples of your replication factor in order to keep the "racks" balanced. In other words, this node should have been added to rack 1, 2 or 3.
Having said that, you should be able to easily fix your problem by running a nodetool repair -pr on the new node. On Sun, Apr 2, 2023 at 8:16 PM David Tinker <david.tin...@gmail.com> wrote: > Hi All > > I recently added a node to my 3 node Cassandra 4.0.5 cluster and now many > reads are not returning rows! What do I need to do to fix this? There > weren't any errors in the logs or other problems that I could see. I > expected the cluster to balance itself but this hasn't happened (yet?). The > nodes are similar so I have num_tokens=256 for each. I am using the > Murmur3Partitioner. > > # nodetool status > Datacenter: dc1 > =============== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns (effective) Host ID > Rack > UN xxx.xxx.xxx.105 2.65 TiB 256 72.9% > afd02287-3f88-4c6f-8b27-06f7a8192402 rack3 > UN xxx.xxx.xxx.253 2.6 TiB 256 73.9% > e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2 > UN xxx.xxx.xxx.24 93.82 KiB 256 80.0% > c4e8b4a0-f014-45e6-afb4-648aad4f8500 rack4 > UN xxx.xxx.xxx.107 2.65 TiB 256 73.2% > ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1 > > # nodetool netstats > Mode: NORMAL > Not sending any streams. > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool Name Active Pending Completed Dropped > Large messages n/a 0 71754 0 > Small messages n/a 0 8398184 14 > Gossip messages n/a 0 1303634 0 > > # nodetool ring > Datacenter: dc1 > ========== > Address Rack Status State Load Owns > Token > > 9189523899826545641 > xxx.xxx.xxx.24 rack4 Up Normal 93.82 KiB 79.95% > -9194674091837769168 > xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB 73.25% > -9168781258594813088 > xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB 73.92% > -9163037340977721917 > xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB 72.88% > -9148860739730046229 > xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB 73.25% > -9125240034139323535 > xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB 73.92% > -9112518853051755414 > xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB 72.88% > -9100516173422432134 > ... > > This is causing a serious production issue. Please help if you can. > > Thanks > David > > > >
