Hello,
While running a cleanup, Cassandra stopped with the following exception and
inspecting the logs revealed several exceptions such as below over the past
3 days.Given that it's dying on compactions, I'm really worried.
If a row was trashed, will the error propagate from node to node or will it
be localized and a repair will fix it ?
Thanks
java.io.IOError:
org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
column name length 0
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
java.io.IOError: java.io.IOException: Corrupt (negative) value length
encountered
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
java.io.IOError: java.io.IOException: mmap segment underflow; remaining is
113362603 but 157851071 requested
at
org.apache.cassandra.io.util.ColumnIterator.deserializeNext(ColumnSortedMap.java:265)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:281)
at
org.apache.cassandra.io.util.ColumnIterator.next(ColumnSortedMap.java:236)
at
java.util.concurrent.ConcurrentSkipListMap.buildFromSorted(ConcurrentSkipListMap.java:1493)
at
java.util.concurrent.ConcurrentSkipListMap.<init>(ConcurrentSkipListMap.java:1443)
at
org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:445)
java.lang.AssertionError: DecoratedKey(Token(bytes[]), ) !=
DecoratedKey(Token(bytes[02c67a8a54000000005311]), 02c67a8a54000000005311)
in /var/lib/cassandra/data/XXXXXX-g-159-Data.db
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1407)
at
org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:1274)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1314)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1261)
at org.apache.cassandra.db.Table.getRow(Table.java:385)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:58)
at
org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:105)
