Hi Robert, Thanks for your response. Replication factor is 3.
We are in the process of upgrading to 2.2.4. We have had too many performance issues with later versions of Cassandra (I have asked asked for help related to that in the forum). We are close to getting to similar performance now and hopefully upgrade in next few weeks. Lot of testing to do :(. We are not removing multiple nodes at the same time. All dead nodes are from same AZ so there were no errors when the nodes were down as expected (because we use QUORUM). However, As soon as I started removing nodes one by one, every time time we see lot of timeout and unavailable exceptions which doesn't make any sense because I am just removing a node that doesn't even exist. From: Robert Coli <rc...@eventbrite.com<mailto:rc...@eventbrite.com>> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Date: Wednesday, March 2, 2016 at 2:52 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <user@cassandra.apache.org<mailto:user@cassandra.apache.org>> Subject: Re: Removing Node causes bunch of HostUnavailableException On Wed, Mar 2, 2016 at 8:10 AM, Peddi, Praveen <pe...@amazon.com<mailto:pe...@amazon.com>> wrote: We have few dead nodes in the cluster (Amazon ASG removed those thinking there is an issue with health). Now we are trying to remove those dead nodes from the cluster so that other nodes can take over. As soon as I execute nodetool removenode <ID>, we see lots of HostUnavailableExceptions both on reads and writes. What I am not able to understand is, these are deadnodes and don't even physically exists. Why would removenode command cause any outage of nodes in Cassandra when we had no errors whatsoever before removing them. I could not really find a jira ticket for this. What is your replication factor? Also, 2.0.9 is meaningfully old at this point, consider upgrading ASAP. Also, removing multiple nodes with removenode means your consistency is pretty hosed. Repair ASAP, but there are potential cases where repair won't help. =Rob =Rob
