Yes replication factor is 3. I ran nodetool repair -pr on all the nodes (one at a time) and am still having issues getting data back from queries.
I did make the new node a seed node. Re "rack4": I assumed that was just an indication as to the physical location of the server for redundancy. This one is separate from the others so I used rack4. On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz <crdiaz...@gmail.com> wrote: > I'm assuming that your replication factor is 3. If that's the case, did > you intentionally put this node in rack 4? Typically, you want to add > nodes in multiples of your replication factor in order to keep the "racks" > balanced. In other words, this node should have been added to rack 1, 2 or > 3. > > Having said that, you should be able to easily fix your problem by running > a nodetool repair -pr on the new node. > > On Sun, Apr 2, 2023 at 8:16 PM David Tinker <david.tin...@gmail.com> > wrote: > >> Hi All >> >> I recently added a node to my 3 node Cassandra 4.0.5 cluster and now many >> reads are not returning rows! What do I need to do to fix this? There >> weren't any errors in the logs or other problems that I could see. I >> expected the cluster to balance itself but this hasn't happened (yet?). The >> nodes are similar so I have num_tokens=256 for each. I am using the >> Murmur3Partitioner. >> >> # nodetool status >> Datacenter: dc1 >> =============== >> Status=Up/Down >> |/ State=Normal/Leaving/Joining/Moving >> -- Address Load Tokens Owns (effective) Host ID >> Rack >> UN xxx.xxx.xxx.105 2.65 TiB 256 72.9% >> afd02287-3f88-4c6f-8b27-06f7a8192402 rack3 >> UN xxx.xxx.xxx.253 2.6 TiB 256 73.9% >> e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2 >> UN xxx.xxx.xxx.24 93.82 KiB 256 80.0% >> c4e8b4a0-f014-45e6-afb4-648aad4f8500 rack4 >> UN xxx.xxx.xxx.107 2.65 TiB 256 73.2% >> ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1 >> >> # nodetool netstats >> Mode: NORMAL >> Not sending any streams. >> Read Repair Statistics: >> Attempted: 0 >> Mismatch (Blocking): 0 >> Mismatch (Background): 0 >> Pool Name Active Pending Completed Dropped >> Large messages n/a 0 71754 0 >> Small messages n/a 0 8398184 14 >> Gossip messages n/a 0 1303634 0 >> >> # nodetool ring >> Datacenter: dc1 >> ========== >> Address Rack Status State Load Owns >> Token >> >> 9189523899826545641 >> xxx.xxx.xxx.24 rack4 Up Normal 93.82 KiB 79.95% >> -9194674091837769168 >> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB 73.25% >> -9168781258594813088 >> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB 73.92% >> -9163037340977721917 >> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB 72.88% >> -9148860739730046229 >> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB 73.25% >> -9125240034139323535 >> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB 73.92% >> -9112518853051755414 >> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB 72.88% >> -9100516173422432134 >> ... >> >> This is causing a serious production issue. Please help if you can. >> >> Thanks >> David >> >> >> >>
