When mutation stage messages are dropped I start by looking for GC problems in the cassandra logs.
You also have some blocked flush writer tasks, which may be due to a large number of CF's, a large number of secondary indexes, slow disk IO or excessive use of snapshot / flush. Check the comments for the memtable_flush_queue_size in the yaml file. Hope that helps. ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 8/05/2013, at 4:41 AM, Keith Wright <kwri...@nanigans.com> wrote: > I am running 1.2.4 with Vnodes and have been writing at low volume. I have > doubled the volume and suddenly 3 of my 6 nodes are showing much higher load > than the others (30 vs 3) and tpstats show the mutation stage as completely > full (see below). I did find a FileNotFoundException that I pasted below > which appears to be caused by creating, dropping, and creating a keyspace > (something I did but 4 or 5 days ago). Anyone have any idea what's going on > here? > > Thanks > > Keiths-MacBook-Pro:bin keith$ ./nodetool tpstats -h lxpcas005.nanigans.com > Pool Name Active Pending Completed Blocked All > time blocked > ReadStage 0 0 130990 0 > 0 > RequestResponseStage 0 0 344216 0 > 0 > MutationStage 128 452 3464036 0 > 0 > ReadRepairStage 0 0 14131 0 > 0 > ReplicateOnWriteStage 0 0 32872 0 > 0 > GossipStage 1 611 6351 0 > 0 > AntiEntropyStage 0 0 0 0 > 0 > MigrationStage 0 0 9 0 > 0 > MemtablePostFlusher 0 0 91 0 > 0 > FlushWriter 0 0 60 0 > 27 > MiscStage 0 0 0 0 > 0 > commitlog_archiver 0 0 0 0 > 0 > InternalResponseStage 0 0 3 0 > 0 > HintedHandoff 1 1 13 0 > 0 > > Message type Dropped > RANGE_SLICE 0 > READ_REPAIR 54 > BINARY 0 > READ 0 > MUTATION 8539 > _TRACE 0 > REQUEST_RESPONSE 0 > > > > ERROR [ReplicateOnWriteStage:95404] 2013-05-06 14:55:06,555 > CassandraDaemon.java (line 174) Exception in thread > Thread[ReplicateOnWriteStage:95404,5,main] > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.FileNotFoundException: > /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db > (No such file or directory) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: > /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db > (No such file or directory) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46) > at > org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57) > at > org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41) > at > org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117) > at > org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64) > at > org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81) > at > org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68) > at > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274) > at > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) > at > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214) > at > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126) > at org.apache.cassandra.db.Table.getRow(Table.java:347) > at > org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64) > at > org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90) > at > org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578) > ... 3 more > Caused by: java.io.FileNotFoundException: > /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db > (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) > at > org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42) > ... 21 more
