I should add that the new node does have some data. On Mon, Apr 3, 2023 at 7:04 AM David Tinker <david.tin...@gmail.com> wrote:
> It is also in the seeds list for the other nodes. Should I remove it from > those, restart them one at a time, then restart it? > > /etc/cassandra # grep -i bootstrap * > doesn't show anything so I don't think I have auto_bootstrap false. > > Thanks very much for the help. > > > On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz <crdiaz...@gmail.com> wrote: > >> Just remove it from the seed list in the cassandra.yaml file and restart >> the node. Make sure that auto_bootstrap is set to true first though. >> >> On Sun, Apr 2, 2023 at 9:59 PM David Tinker <david.tin...@gmail.com> >> wrote: >> >>> So likely because I made it a seed node when I added it to the cluster >>> it didn't do the bootstrap process. How can I recover this? >>> >>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker <david.tin...@gmail.com> >>> wrote: >>> >>>> Yes replication factor is 3. >>>> >>>> I ran nodetool repair -pr on all the nodes (one at a time) and am >>>> still having issues getting data back from queries. >>>> >>>> I did make the new node a seed node. >>>> >>>> Re "rack4": I assumed that was just an indication as to the physical >>>> location of the server for redundancy. This one is separate from the others >>>> so I used rack4. >>>> >>>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz <crdiaz...@gmail.com> wrote: >>>> >>>>> I'm assuming that your replication factor is 3. If that's the case, >>>>> did you intentionally put this node in rack 4? Typically, you want to add >>>>> nodes in multiples of your replication factor in order to keep the "racks" >>>>> balanced. In other words, this node should have been added to rack 1, 2 >>>>> or >>>>> 3. >>>>> >>>>> Having said that, you should be able to easily fix your problem by >>>>> running a nodetool repair -pr on the new node. >>>>> >>>>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker <david.tin...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi All >>>>>> >>>>>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and now >>>>>> many reads are not returning rows! What do I need to do to fix this? >>>>>> There >>>>>> weren't any errors in the logs or other problems that I could see. I >>>>>> expected the cluster to balance itself but this hasn't happened (yet?). >>>>>> The >>>>>> nodes are similar so I have num_tokens=256 for each. I am using the >>>>>> Murmur3Partitioner. >>>>>> >>>>>> # nodetool status >>>>>> Datacenter: dc1 >>>>>> =============== >>>>>> Status=Up/Down >>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>> -- Address Load Tokens Owns (effective) Host ID >>>>>> Rack >>>>>> UN xxx.xxx.xxx.105 2.65 TiB 256 72.9% >>>>>> afd02287-3f88-4c6f-8b27-06f7a8192402 rack3 >>>>>> UN xxx.xxx.xxx.253 2.6 TiB 256 73.9% >>>>>> e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2 >>>>>> UN xxx.xxx.xxx.24 93.82 KiB 256 80.0% >>>>>> c4e8b4a0-f014-45e6-afb4-648aad4f8500 rack4 >>>>>> UN xxx.xxx.xxx.107 2.65 TiB 256 73.2% >>>>>> ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1 >>>>>> >>>>>> # nodetool netstats >>>>>> Mode: NORMAL >>>>>> Not sending any streams. >>>>>> Read Repair Statistics: >>>>>> Attempted: 0 >>>>>> Mismatch (Blocking): 0 >>>>>> Mismatch (Background): 0 >>>>>> Pool Name Active Pending Completed Dropped >>>>>> Large messages n/a 0 71754 0 >>>>>> Small messages n/a 0 8398184 14 >>>>>> Gossip messages n/a 0 1303634 0 >>>>>> >>>>>> # nodetool ring >>>>>> Datacenter: dc1 >>>>>> ========== >>>>>> Address Rack Status State Load Owns >>>>>> Token >>>>>> >>>>>> 9189523899826545641 >>>>>> xxx.xxx.xxx.24 rack4 Up Normal 93.82 KiB >>>>>> 79.95% -9194674091837769168 >>>>>> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB >>>>>> 73.25% -9168781258594813088 >>>>>> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB >>>>>> 73.92% -9163037340977721917 >>>>>> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB >>>>>> 72.88% -9148860739730046229 >>>>>> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB >>>>>> 73.25% -9125240034139323535 >>>>>> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB >>>>>> 73.92% -9112518853051755414 >>>>>> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB >>>>>> 72.88% -9100516173422432134 >>>>>> ... >>>>>> >>>>>> This is causing a serious production issue. Please help if you can. >>>>>> >>>>>> Thanks >>>>>> David >>>>>> >>>>>> >>>>>> >>>>>>
