You can drop the hints via JMX and stopping the node and deleting the SSTables.
Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 25/04/2013, at 12:27 AM, Michal Michalski <mich...@opera.com> wrote: > Not really sure if it has something to do with the schema problems, but I > think the fact that node was down caused us to hit > https://issues.apache.org/jira/browse/CASSANDRA-5179 (a bit different output > on sender's side, but looks similar in general) - after checking logs with > debug level TRACE it seems that all these "add mutation & compact forever" is > caused by "infinite" HH from some of the nodes. Even if it's not a root > cause, it still a little step forward ;-) > > BTW. Looks eeevil! > > cqlsh:system> SELECT * FROM hints limit 1; > Request did not complete within rpc_timeout. > > root@cssa02-13:~# ls -lahS /cassandra/system/hints/ | grep "Data\." | wc -l > 83 > > I don't even want to think what's inside ;-) > > M. > > W dniu 24.04.2013 08:54, Michal Michalski pisze: >> >>> The log messages seem fine to me. It's handling eventually updating >>> the secondary indexes. >> >> Yes, but now it also applies mutations, while (I'm 100% sure) nothing >> was written to the affected CFs ;-) >> >> DEBUG [MutationStage:87] 2013-04-24 06:41:45,695 >> RowMutationVerbHandler.java (line 40) Applying mutation >> DEBUG [MutationStage:56] 2013-04-24 06:41:45,695 >> RowMutationVerbHandler.java (line 40) Applying mutation >> DEBUG [MutationStage:123] 2013-04-24 06:41:45,695 >> RowMutationVerbHandler.java (line 40) Applying mutation >> DEBUG [MutationStage:118] 2013-04-24 06:41:45,695 >> >> And it's happening for > 24 hours now with about 1000 mutations / sec >> (with no writes from clients!), for only 3 CFs (and as I said, this CFs >> are between 2 and 80 MB large), causing non-stop SSTable >> creation/compaction - see attached screen. >> >> Strange, but at least the node is more or less "fine" in general (and >> the second one is 100% fine) ;-) >> >> Anyway, my original schema problem is still there and I can't update >> anything, but I'm not going to make an attempt to fix it once again 2 >> days before my holidays, so I think I'll get back to this problem in the >> end of May ;-) >> >> Thanks for help :-) >> >> M. >> >
