Running Cassandra 1.2.9 in AWS with a 12 host cluster, I am getting lots of
CorruptSSTableException in system.log on one of my hosts.
Is it possible to find out which SSTable(s) is/are corrupt?
I'm currently running "nodetool scrub" on the relevant host, but that
doesn't seem like an efficient way to fix the problem (if it fixes it at
all).
Here's an example error:
ERROR [ReplicateOnWriteStage:1070] 2014-02-25 17:43:03,518
CassandraDaemon.java (line 192) Exception in thread
Thread[ReplicateOnWriteStage:1070,5,main]
java.lang.RuntimeException:
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.EOFException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1597)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.EOFException
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:65)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:272)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1391)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at
org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:772)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1593)
... 3 more
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:380)
at
org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:371)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:116)
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:60)
... 15 more
