FWIW, `nodetool decommission` is strongly preferred. `nodetool removenode` is designed to be run when a host is offline. Only decommission is guaranteed to maintain consistency / correctness, and removemode probably streams a lot more data around than decommission.
On Mon, Apr 3, 2023 at 6:47 AM Bowen Song via user < user@cassandra.apache.org> wrote: > Use nodetool removenode is strongly preferred in most circumstances, and > only resort to assassinate if you do not care about data consistency or > you know there won't be any consistency issue (e.g. no new writes and did > not run nodetool cleanup). > > Since the size of data on the new node is small, nodetool removenode > should finish fairly quickly and bring your cluster back. > > Next time when you are doing something like this again, please test it out > on a non-production environment, make sure everything works as expected > before moving onto the production. > > > On 03/04/2023 06:28, David Tinker wrote: > > Should I use assassinate or removenode? Given that there is some data on > the node. Or will that be found on the other nodes? Sorry for all the > questions but I really don't want to mess up. > > On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz <crdiaz...@gmail.com> wrote: > >> That's what nodetool assassinte will do. >> >> On Sun, Apr 2, 2023 at 10:19 PM David Tinker <david.tin...@gmail.com> >> wrote: >> >>> Is it possible for me to remove the node from the cluster i.e. to undo >>> this mess and get the cluster operating again? >>> >>> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz <crdiaz...@gmail.com> wrote: >>> >>>> You can leave it in the seed list of the other nodes, just make sure >>>> it's not included in this node's seed list. However, if you do decide to >>>> fix the issue with the racks first assassinate this node (nodetool >>>> assassinate <ip>), and update the rack name before you restart. >>>> >>>> On Sun, Apr 2, 2023 at 10:06 PM David Tinker <david.tin...@gmail.com> >>>> wrote: >>>> >>>>> It is also in the seeds list for the other nodes. Should I remove it >>>>> from those, restart them one at a time, then restart it? >>>>> >>>>> /etc/cassandra # grep -i bootstrap * >>>>> doesn't show anything so I don't think I have auto_bootstrap false. >>>>> >>>>> Thanks very much for the help. >>>>> >>>>> >>>>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz <crdiaz...@gmail.com> >>>>> wrote: >>>>> >>>>>> Just remove it from the seed list in the cassandra.yaml file and >>>>>> restart the node. Make sure that auto_bootstrap is set to true first >>>>>> though. >>>>>> >>>>>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker <david.tin...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> So likely because I made it a seed node when I added it to the >>>>>>> cluster it didn't do the bootstrap process. How can I recover this? >>>>>>> >>>>>>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker <david.tin...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Yes replication factor is 3. >>>>>>>> >>>>>>>> I ran nodetool repair -pr on all the nodes (one at a time) and am >>>>>>>> still having issues getting data back from queries. >>>>>>>> >>>>>>>> I did make the new node a seed node. >>>>>>>> >>>>>>>> Re "rack4": I assumed that was just an indication as to the >>>>>>>> physical location of the server for redundancy. This one is separate >>>>>>>> from >>>>>>>> the others so I used rack4. >>>>>>>> >>>>>>>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz <crdiaz...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I'm assuming that your replication factor is 3. If that's the >>>>>>>>> case, did you intentionally put this node in rack 4? Typically, you >>>>>>>>> want >>>>>>>>> to add nodes in multiples of your replication factor in order to keep >>>>>>>>> the >>>>>>>>> "racks" balanced. In other words, this node should have been added >>>>>>>>> to rack >>>>>>>>> 1, 2 or 3. >>>>>>>>> >>>>>>>>> Having said that, you should be able to easily fix your problem by >>>>>>>>> running a nodetool repair -pr on the new node. >>>>>>>>> >>>>>>>>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker < >>>>>>>>> david.tin...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi All >>>>>>>>>> >>>>>>>>>> I recently added a node to my 3 node Cassandra 4.0.5 cluster and >>>>>>>>>> now many reads are not returning rows! What do I need to do to fix >>>>>>>>>> this? >>>>>>>>>> There weren't any errors in the logs or other problems that I could >>>>>>>>>> see. I >>>>>>>>>> expected the cluster to balance itself but this hasn't happened >>>>>>>>>> (yet?). The >>>>>>>>>> nodes are similar so I have num_tokens=256 for each. I am using the >>>>>>>>>> Murmur3Partitioner. >>>>>>>>>> >>>>>>>>>> # nodetool status >>>>>>>>>> Datacenter: dc1 >>>>>>>>>> =============== >>>>>>>>>> Status=Up/Down >>>>>>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>>>>>> -- Address Load Tokens Owns (effective) Host ID >>>>>>>>>> Rack >>>>>>>>>> UN xxx.xxx.xxx.105 2.65 TiB 256 72.9% >>>>>>>>>> afd02287-3f88-4c6f-8b27-06f7a8192402 rack3 >>>>>>>>>> UN xxx.xxx.xxx.253 2.6 TiB 256 73.9% >>>>>>>>>> e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2 >>>>>>>>>> UN xxx.xxx.xxx.24 93.82 KiB 256 80.0% >>>>>>>>>> c4e8b4a0-f014-45e6-afb4-648aad4f8500 rack4 >>>>>>>>>> UN xxx.xxx.xxx.107 2.65 TiB 256 73.2% >>>>>>>>>> ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1 >>>>>>>>>> >>>>>>>>>> # nodetool netstats >>>>>>>>>> Mode: NORMAL >>>>>>>>>> Not sending any streams. >>>>>>>>>> Read Repair Statistics: >>>>>>>>>> Attempted: 0 >>>>>>>>>> Mismatch (Blocking): 0 >>>>>>>>>> Mismatch (Background): 0 >>>>>>>>>> Pool Name Active Pending Completed >>>>>>>>>> Dropped >>>>>>>>>> Large messages n/a 0 71754 >>>>>>>>>> 0 >>>>>>>>>> Small messages n/a 0 8398184 >>>>>>>>>> 14 >>>>>>>>>> Gossip messages n/a 0 1303634 >>>>>>>>>> 0 >>>>>>>>>> >>>>>>>>>> # nodetool ring >>>>>>>>>> Datacenter: dc1 >>>>>>>>>> ========== >>>>>>>>>> Address Rack Status State Load >>>>>>>>>> Owns Token >>>>>>>>>> >>>>>>>>>> 9189523899826545641 >>>>>>>>>> xxx.xxx.xxx.24 rack4 Up Normal 93.82 KiB >>>>>>>>>> 79.95% -9194674091837769168 >>>>>>>>>> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB >>>>>>>>>> 73.25% -9168781258594813088 >>>>>>>>>> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB >>>>>>>>>> 73.92% -9163037340977721917 >>>>>>>>>> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB >>>>>>>>>> 72.88% -9148860739730046229 >>>>>>>>>> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB >>>>>>>>>> 73.25% -9125240034139323535 >>>>>>>>>> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB >>>>>>>>>> 73.92% -9112518853051755414 >>>>>>>>>> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB >>>>>>>>>> 72.88% -9100516173422432134 >>>>>>>>>> ... >>>>>>>>>> >>>>>>>>>> This is causing a serious production issue. Please help if you >>>>>>>>>> can. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> David >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>
