Morning,
I'm getting the following error (21 node cluster running 1.2.8)
FSReadError in
/var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:93)
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:312)
at
org.apache.cassandra.io.sstable.SSTableScanner.seekTo(SSTableScanner.java:98)
at
org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:73)
at
org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1424)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1476)
at
org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:46)
at
org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:58)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Input/output error
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
at sun.nio.ch.IOUtil.read(IOUtil.java:198)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:143)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:108)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:85)
... 11 more
This starts during compaction and compaction ends up stuck, the node
eventually winds up locked up with this error repeating throughout the
logs, and requires a restart.
On restart I tried running a nodetool scrub against the offending
column family, which itself failed.
Any suggestions on taking care of this? Should I just delete the
sstable and repair? The node keeps starting compactions and running
into this after a period of time.
David
