Re: mutation stalls and FileNotFoundException

2013-05-09 Thread aaron morton
When mutation stage messages are dropped I start by looking for GC problems in the cassandra logs. You also have some blocked flush writer tasks, which may be due to a large number of CF's, a large number of secondary indexes, slow disk IO or excessive use of snapshot / flush. Check the comment

mutation stalls and FileNotFoundException

2013-05-07 Thread Keith Wright
I am running 1.2.4 with Vnodes and have been writing at low volume. I have doubled the volume and suddenly 3 of my 6 nodes are showing much higher load than the others (30 vs 3) and tpstats show the mutation stage as completely full (see below). I did find a FileNotFoundException that I pasted