To be clear, I had to interrupt a clean operation earlier in the day be killing the cassandra process. Now the node works for awhile, but continually logging the "Error in row mutation" errors then eventually logs a "Fatal exception in thread" error. After which, the process stays alive but there seem to be problems reading from the node. At the very least, read performance is massively degraded.
On Sat, Dec 4, 2010 at 5:52 PM, Dan Hendry <dan.hendry.j...@gmail.com>wrote: > One of my Cassandra nodes is giving me a number of errors then effectively > dying. I think it was somehow caused by interrupting a nodetool clean > operation. Running a recent 0.7 build out of svn. > > ERROR [MutationStage:26] 2010-12-04 16:23:04,395 > RowMutationVerbHandler.java (line 83) Error in row mutation > java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at > org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:264) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363) > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:312) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129) > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120) > at > org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383) > at > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393) > at > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [MutationStage:13] 2010-12-04 16:25:04,061 > RowMutationVerbHandler.java (line 83) Error in row mutation > org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find > cfId=524288 > at > org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:117) > at > org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383) > at > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393) > at > org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [MutationStage:20] 2010-12-04 16:25:25,216 > DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor > java.lang.NullPointerException > at org.apache.cassandra.db.Table.apply(Table.java:398) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:73) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [MutationStage:20] 2010-12-04 16:25:25,216 > AbstractCassandraDaemon.java (line 90) Fatal exception in thread > Thread[MutationStage:20,5,main] > java.lang.NullPointerException > at org.apache.cassandra.db.Table.apply(Table.java:398) > at > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:73) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > ERROR [COMMIT-LOG-WRITER] 2010-12-04 16:25:25,216 > AbstractCassandraDaemon.java (line 90) Fatal exception in thread > Thread[COMMIT-LOG-WRITER,5,main] > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.NullPointerException > at > org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:92) > at > org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:509) > at > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > ... 1 more > >
