Hi ,
We are using apache cassandra 1.2.13 version with three nodes. In that with
a high load we are getting following exception.[1] .
Could some one help on this. This is already reported in [2]
[1]
INFO [ScheduledTasks:1] 2014-02-27 21:56:59,928 GCInspector.java (line 119)
GC for ParNew: 241 ms for 1 collections, 1191010416 used; max is 8375238656
INFO [MemoryMeter:1] 2014-02-27 21:57:38,322 Memtable.java (line 516)
CFS(Keyspace='QpidKeySpace', ColumnFamily='QueueEntries') liveRatio is
49.411764705882355 (just-counted was 49.411764705882355). calculation took
0ms for 1 columns
ERROR [ReadStage:923] 2014-02-27 22:01:42,284 CassandraDaemon.java (line
191) Exception in thread Thread[ReadStage:923,5,main]
java.lang.RuntimeException:
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.IOException: Corrupt (negative) value length encountered
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(
StorageProxy.java:1614)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.IOException: Corrupt (negative) value length encountered
at org.apache.cassandra.db.columniterator.IndexedSliceReader$
IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:357)
at org.apache.cassandra.db.columniterator.IndexedSliceReader.
computeNext(IndexedSliceReader.java:166)
at org.apache.cassandra.db.columniterator.IndexedSliceReader.
computeNext(IndexedSliceReader.java:50)
at com.google.common.collect.AbstractIterator.tryToComputeNext(
AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(
AbstractIterator.java:138)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(
SSTableSliceIterator.java:90)
at org.apache.cassandra.db.filter.QueryFilter$2.getNext(
QueryFilter.java:171)
at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(
QueryFilter.java:154)
at org.apache.cassandra.utils.MergeIterator$Candidate.
advance(MergeIterator.java:143)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.
advance(MergeIterator.java:122)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.
computeNext(MergeIterator.java:96)
at com.google.common.collect.AbstractIterator.tryToComputeNext(
AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(
AbstractIterator.java:138)
at org.apache.cassandra.db.filter.SliceQueryFilter.
collectReducedColumns(SliceQueryFilter.java:160)
at org.apache.cassandra.db.filter.QueryFilter.
collateColumns(QueryFilter.java:136)
at org.apache.cassandra.db.filter.QueryFilter.
collateOnDiskAtom(QueryFilter.java:84)
at org.apache.cassandra.db.CollationController.collectAllData(
CollationController.java:291)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(
CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(
ColumnFamilyStore.java:1397)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(
ColumnFamilyStore.java:1213)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(
ColumnFamilyStore.java:1129)
at org.apache.cassandra.db.Table.getRow(Table.java:344)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(
SliceFromReadCommand.java:70)
at org.apache.cassandra.service.StorageProxy$
LocalReadRunnable.runMayThrow(StorageProxy.java:1058)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(
StorageProxy.java:1610)
... 3 more
Caused by: java.io.IOException: Corrupt (negative) value length encountered
at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(
ByteBufferUtil.java:352)
at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(
ColumnSerializer.java:102)
at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(
OnDiskAtom.java:92)
at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(
OnDiskAtom.java:73)
at org.apache.cassandra.db.columniterator.IndexedSliceReader$
IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:398)
at org.apache.cassandra.db.columniterator.IndexedSliceReader$
IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:353)
... 27 more
[2]https://issues.apache.org/jira/browse/CASSANDRA-6536
--
Best Regards,
* Shammi Jayasinghe*
Associate Tech Lead
WSO2, Inc.; http://wso2.com,
mobile: +94 71 4493085
