Just to be sure: you have only one datacenter configured in Cassandra? Hannu
> On 27 Jun 2017, at 0.02, Rutvij Bhatt <rut...@sense.com> wrote: > > Hi guys, > > I observed some odd behaviour with our Cassandra cluster the other day while > doing some maintenance operation and was wondering if anyone would be able to > provide some insight. > > Initially, I started a node up to join the cluster. That node appeared to be > having issues joining due to some SSTable corruption it encountered. Since it > was still in early staged and I had never seen this failure before, I decided > to take it out of commission and just try again. However, since it was in a > bad state, I decided to issue a "nodetool removenode <host id>" on a peer > rather than a "nodetool decommission" on the node itself. > > The removenode command hung indefinitely - my guess is that this is related > to https://issues.apache.org/jira/browse/CASSANDRA-6542. We are using 2.1.11. > > While this was happening, the driver in the application started logging error > messages about not being able to reach a quorum of 4. This, to me, was > mysterious as none of my keyspaces have an RF > 3. That quorum count in the > error implied an RF of 6 or 7. > > I eventually forced that node out of the ring with "nodetool removenode > force". This seemed to mostly fix the issue, though there seems to have been > enough of a load spike to cause some of the machines' JVMs to accumulate a > lot of garbage very fast and spit out a ton of "Not marking nodes down due to > local pause of ... ", trying to clean it up. Some of these nodes seemed > unresponsive to their peers, who marked them DOWN (as indicated by "nodetool > status" and the cassandra log file on those machines), further exacerbating > the situation on the nodes that were still up. > > I guess my question is two-fold. First, can anyone provide some insight into > what may have happened? Second, what do you consider good practices when > dealing with such issues? Any advice is greatly appreciated! > > Thanks, > Rutvij
