Hi guys, just for the record, in case someone has this issue in the future, it is a bug, fixed in 1.2.12.
I recommend anyone in this version to upgrade cluster before getting totally stuck (by adding new machines / DC / altering keyspaces / ...), well anything using gossip actually. We are going to plan a full downtime to be able to recover, since we are completely stuck. Datastax help to find out the issue, thanks DuyHai ! For more information => https://issues.apache.org/jira/browse/CASSANDRA-6297 2014-06-26 16:16 GMT+02:00 Jonathan Lacefield <jlacefi...@datastax.com>: > Hello Alain, > > I'm not sure of the root cause of this item. It may be helpful to use > DEBUG and start the node to see what's happening as well as watch > compaction stats or tpstats to understand what is taxing your system. > > The log file you provided shows a large ParNew while replaying commit > log segments. Does your app insert very large "rows" or have individual > columns that are large? > > I quickly reviewed Changes/txt > https://github.com/apache/cassandra/blob/cassandra-1.2/CHANGES.txt to see > if anything jumps out as a culprit, but didn't spot anything. > > Sorry i can't be of more help with this one. It may take some hands-on > investigation or maybe someone else in the community has experienced this > issue and can provide feedback. > > Thanks, > > Jonathan > > Jonathan Lacefield > Solutions Architect, DataStax > (404) 822 3487 > <http://www.linkedin.com/in/jlacefield> > > <http://www.datastax.com/cassandrasummit14> > > > > On Wed, Jun 18, 2014 at 3:07 PM, Robert Coli <rc...@eventbrite.com> wrote: > >> On Wed, Jun 18, 2014 at 5:36 AM, Alain RODRIGUEZ <arodr...@gmail.com> >> wrote: >> >>> We stop the node using : nodetool disablegossip && nodetool >>> disablethrift && nodetool disablebinary && sleep 10 && nodetool drain && >>> sleep 30 && service cassandra stop >>> >> >> The stuff before "nodetool drain" here is redundant and doesn't actually >> do what you are expecting it to do. >> >> https://issues.apache.org/jira/browse/CASSANDRA-4162 >> >> =Rob >> >> > >
