Hi All
I recently added a node to my 3 node Cassandra 4.0.5 cluster and now many
reads are not returning rows! What do I need to do to fix this? There
weren't any errors in the logs or other problems that I could see. I
expected the cluster to balance itself but this hasn't happened (yet?). The
nodes are similar so I have num_tokens=256 for each. I am using the
Murmur3Partitioner.
# nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN xxx.xxx.xxx.105 2.65 TiB 256 72.9%
afd02287-3f88-4c6f-8b27-06f7a8192402 rack3
UN xxx.xxx.xxx.253 2.6 TiB 256 73.9%
e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2
UN xxx.xxx.xxx.24 93.82 KiB 256 80.0%
c4e8b4a0-f014-45e6-afb4-648aad4f8500 rack4
UN xxx.xxx.xxx.107 2.65 TiB 256 73.2%
ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1
# nodetool netstats
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name Active Pending Completed Dropped
Large messages n/a 0 71754 0
Small messages n/a 0 8398184 14
Gossip messages n/a 0 1303634 0
# nodetool ring
Datacenter: dc1
==========
Address Rack Status State Load Owns
Token
9189523899826545641
xxx.xxx.xxx.24 rack4 Up Normal 93.82 KiB 79.95%
-9194674091837769168
xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB 73.25%
-9168781258594813088
xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB 73.92%
-9163037340977721917
xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB 72.88%
-9148860739730046229
xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB 73.25%
-9125240034139323535
xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB 73.92%
-9112518853051755414
xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB 72.88%
-9100516173422432134
...
This is causing a serious production issue. Please help if you can.
Thanks
David
