Many files are corrupted when our cassandra is update to 0.6.2
COMPACTION-POOL is down caused by the following error.
and some nodes can NOT startup because of this error.
Is it caused by the issue CASSANDRA-1169? The node got the wrong or corrupted
stream file?
ERROR [COMPACTION-POOL:1] 2010-06-12 11:28:40,279
DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
java.util.concurrent.ExecutionException:
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of
range: -1
at java.lang.String.substring(String.java:1937)
at
org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:50)
at org.apache.cassandra.io.IteratingRow.<init>(IteratingRow.java:48)
at
org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:136)
at
org.apache.cassandra.io.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:113)
at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:105)
at org.apache.cassandra.io.SSTableScanner.next(SSTableScanner.java:34)
at
org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284)
at
org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326)
at
org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:284)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
INFO [main] 2010-06-12 10:38:43,764 SSTableReader.java (line 124) Sampling
index for /data4/cassandra2/data/GroupChat/GroupTopTopic-839-Data.db
ERROR [main] 2010-06-12 10:38:43,777 CassandraDaemon.java (line 195) Exception
encountered during startup.
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1937)
at
org.apache.cassandra.dht.RandomPartitioner.convertFromDiskFormat(RandomPartitioner.java:50)
at
org.apache.cassandra.io.SSTableReader.loadIndexFile(SSTableReader.java:261)
at org.apache.cassandra.io.SSTableReader.open(SSTableReader.java:125)
at org.apache.cassandra.io.SSTableReader.open(SSTableReader.java:114)
at
org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:178)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:248)
at org.apache.cassandra.db.Table.<init>(Table.java:338)
at org.apache.cassandra.db.Table.open(Table.java:199)
at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:91)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
INFO [main] 2010-06-12 10:45:12,507 SSTableReader.java (line 124) Sampling
index for /data4/cassandra2/data/GroupChat/Topic-1236-Data.db
ERROR [main] 2010-06-12 10:45:12,528 ColumnFamilyStore.java (line 182) Corrupt
file /data4/cassandra2/data/GroupChat/Topic-1236-Data.db; skipped
java.io.StreamCorruptedException: invalid stream header: 32393031
at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:783)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
at
org.apache.cassandra.utils.BitSetSerializer.deserialize(BitSetSerializer.java:41)
at
org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilter.java:207)
at
org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilter.java:195)
at
org.apache.cassandra.io.SSTableReader.loadBloomFilter(SSTableReader.java:236)
at org.apache.cassandra.io.SSTableReader.open(SSTableReader.java:126)
at org.apache.cassandra.io.SSTableReader.open(SSTableReader.java:114)
at
org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:178)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:248)
at org.apache.cassandra.db.Table.<init>(Table.java:338)
at org.apache.cassandra.db.Table.open(Table.java:199)
at
org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:91)
at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:177)
ERROR [ROW-READ-STAGE:87] 2010-06-12 10:08:16,334
DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
ERROR [ROW-READ-STAGE:87] 2010-06-12 10:08:16,335 CassandraDaemon.java (line
78) Fatal exception in thread Thread[ROW-READ-STAGE:87,5,main]
ERROR [ROW-READ-STAGE:79] 2010-06-12 10:09:06,606
DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
ERROR [ROW-READ-STAGE:79] 2010-06-12 10:09:06,607 CassandraDaemon.java (line
78) Fatal exception in thread Thread[ROW-READ-STAGE:79,5,main]
ERROR [ROW-READ-STAGE:54] 2010-06-12 10:22:42,623
DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
ERROR [ROW-READ-STAGE:54] 2010-06-12 10:22:42,623 CassandraDaemon.java (line
78) Fatal exception in thread Thread[ROW-READ-STAGE:54,5,main]
ERROR [ROW-READ-STAGE:86] 2010-06-12 10:22:42,873
DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
ERROR [ROW-READ-STAGE:86] 2010-06-12 10:22:42,874 CassandraDaemon.java (line
78) Fatal exception in thread Thread[ROW-READ-STAGE:86,5,main]
