Hello, please see these issues: https://issues.apache.org/jira/browse/CASSANDRA-5686 and https://issues.apache.org/jira/browse/CASSANDRA-5391 if you hit any of them.
regards, ondrej cernos On Wed, Aug 7, 2013 at 5:00 PM, Madalina Matei <madalinaima...@gmail.com>wrote: > Hi, > > I have a 5 nodes cassandra (version 1.1.5) ring, RF=2, CL- READ/Write =1. > After a node went down without any error reported in OS syslog or Cassandra > syslog i decided to perform a repair. > > Each time i run a nodetool repair I get this error: > > INFO [FlushWriter:5] 2013-08-07 11:09:26,770 Memtable.java (line 305) > Completed flushing /data/<data>-298-Data.db (18694 bytes) for commitlog > position ReplayPosition(segmentId=1375867548785, position=199) > ERROR [Thrift:286] 2013-08-07 11:10:04,448 CustomTThreadPoolServer.java > (line 204) Error occurred during processing of message. > java.lang.RuntimeException: error reading 1 of 1 > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83) > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:39) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:116) > at > org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:203) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:117) > at > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:140) > at > org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:107) > at > org.apache.cassandra.db.RowIteratorFactory$2.getReduced(RowIteratorFactory.java:80) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381) > at > org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433) > at > org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50) > at > org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870) > at > org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691) > at > org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008) > at > org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5) > at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) > at org.xerial.snappy.SnappyNative.rawUncompress(Native Method) > at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391) > at > org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361) > at > org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:88) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:83) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:73) > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37) > at > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:79) > ... 32 more > > > I performed first a sstablescrub and i got the same error. > > The problem is that my map-reduce job fails each time throwing in this > specific node syslog an error complaining about the same SSTable. > > ERROR [Thrift:50] 2013-08-07 09:30:40,103 CustomTThreadPoolServer.java > (line 204) Error occurred during processing of message. > java.lang.RuntimeException: SSTableScanner(file=/data/<data>-298-Data.db- > chunk length 65536, data length 773855911. > sstable=SSTableReader(path='/data/<data>-298-Data.db') exhausted=false) > failed to provide next columns from > KeyScanningIterator(finishedAt:105054114) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:198) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:151) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:143) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:149) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:126) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:100) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1381) > at > org.apache.cassandra.db.ColumnFamilyStore$2.computeNext(ColumnFamilyStore.java:1377) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1454) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1433) > at > org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:50) > at > org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870) > at > org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:691) > at > org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3008) > at > org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:2996) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5) > at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) > at org.xerial.snappy.SnappyNative.rawUncompress(Native Method) > at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391) > at > org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:94) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:91) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:77) > at > org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381) > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361) > at > org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324) > at > org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398) > at > org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:178) > > > Is there any way to fix that specific SSTable or any other recommendations? > > Thanks, > Madalina > >
