Not really sure if it has something to do with the schema problems, but
I think the fact that node was down caused us to hit
https://issues.apache.org/jira/browse/CASSANDRA-5179 (a bit different
output on sender's side, but looks similar in general) - after checking
logs with debug level TRACE it seems that all these "add mutation &
compact forever" is caused by "infinite" HH from some of the nodes. Even
if it's not a root cause, it still a little step forward ;-)
BTW. Looks eeevil!
cqlsh:system> SELECT * FROM hints limit 1;
Request did not complete within rpc_timeout.
root@cssa02-13:~# ls -lahS /cassandra/system/hints/ | grep "Data\." | wc -l
83
I don't even want to think what's inside ;-)
M.
W dniu 24.04.2013 08:54, Michal Michalski pisze:
The log messages seem fine to me. It's handling eventually updating
the secondary indexes.
Yes, but now it also applies mutations, while (I'm 100% sure) nothing
was written to the affected CFs ;-)
DEBUG [MutationStage:87] 2013-04-24 06:41:45,695
RowMutationVerbHandler.java (line 40) Applying mutation
DEBUG [MutationStage:56] 2013-04-24 06:41:45,695
RowMutationVerbHandler.java (line 40) Applying mutation
DEBUG [MutationStage:123] 2013-04-24 06:41:45,695
RowMutationVerbHandler.java (line 40) Applying mutation
DEBUG [MutationStage:118] 2013-04-24 06:41:45,695
And it's happening for > 24 hours now with about 1000 mutations / sec
(with no writes from clients!), for only 3 CFs (and as I said, this CFs
are between 2 and 80 MB large), causing non-stop SSTable
creation/compaction - see attached screen.
Strange, but at least the node is more or less "fine" in general (and
the second one is 100% fine) ;-)
Anyway, my original schema problem is still there and I can't update
anything, but I'm not going to make an attempt to fix it once again 2
days before my holidays, so I think I'll get back to this problem in the
end of May ;-)
Thanks for help :-)
M.
