Re: Large number of pending gossip stage tasks in nodetool tpstats

2013-08-07 Thread Faraaz Sareshwala
And by that last statement, I mean are there any further things I should look for given the information in my response? I'll definitely look at implementing your suggestions and see what I can find. On Aug 7, 2013, at 7:31 PM, "Faraaz Sareshwala" wrote: > Thanks Aaron. The node that was behav

Re: Large number of pending gossip stage tasks in nodetool tpstats

2013-08-07 Thread Faraaz Sareshwala
Thanks Aaron. The node that was behaving this way was a production node so I had to take some drastic measures to get it back to doing the right thing. It's no longer behaving this way after wiping the system tables and having cassandra resync the schema from other nodes. In hindsight, maybe I c

Re: Large number of pending gossip stage tasks in nodetool tpstats

2013-08-07 Thread Aaron Morton
> When looking at nodetool > gossipinfo, I notice that this node has updated to the latest schema hash, but > that it thinks other nodes in the cluster are on the older version. What does describe cluster in cassandra-cli say ? It will let you know if there are multiple schema versions in the clu

Large number of pending gossip stage tasks in nodetool tpstats

2013-08-06 Thread Faraaz Sareshwala
I'm running cassandra-1.2.8 in a cluster with 45 nodes across three racks. All nodes are well behaved except one. Whenever I start this node, it starts churning CPU. Running nodetool tpstats, I notice that the number of pending gossip stage tasks is constantly increasing [1]. When looking at nodeto