The sha1 don't match, which would indicate that the sstable has been modified after being written. But Cassandra never modify a sstable once it has been written, so this would suggest an external modification, typically some bit rot.
In that case you don't have much other choice than removing the mentioned data file and run a repair. -- Sylvain On Fri, Dec 9, 2011 at 1:37 PM, Ramesh Natarajan <rames...@gmail.com> wrote: > Hi, > > I have a 30 node cassandra cluster running on RHEL6 64 bit. RF=3, > reads and writes performed with QUORUM. After few hours of test run, > I am seeing this error in the system.log file. > > [root@MSA-VM-18 cassandra]# cat > /var/lib/cassandra/data/MSA/modseq-hb-419-Digest.sha1 > 71e43a932a29553720149bb4f93727e4d269735d > modseq-hb-419-Data.db[root@MSA-VM-18 cassandra]# > [root@MSA-VM-18 cassandra]# sha1sum > /var/lib/cassandra/data/MSA/modseq-hb-419-Data.db > 033f5aea5590851377d3bb79df27f0e6eedb6b95 > /var/lib/cassandra/data/MSA/modseq-hb-419-Data.db > [root@MSA-VM-18 cassandra]# > > Any pointers to troubleshoot this issue? > > I am attaching the system.log file for your reference. > > thanks > Ramesh > > > INFO [CompactionExecutor:296] 2011-12-09 04:36:40,430 > CompactionTask.java (line 112) Compacting > [SSTableReader(path='/var/lib/cassandra/data/MSA/transactions-hb-55-Data.db'), > SSTableReader(path='/var/lib/cassandra/data/MSA/transactions-hb-53-Data.db')] > INFO [CompactionExecutor:296] 2011-12-09 04:36:40,501 > CompactionTask.java (line 213) Compacted to > [/var/lib/cassandra/data/MSA/transactions-hb-56-Data.db,]. 280,210 to > 144,785 (~51% of original) bytes for 3 keys at 2.191710MB/s. Time: > 63ms. > ERROR [CompactionExecutor:295] 2011-12-09 04:36:41,320 > AbstractCassandraDaemon.java (line 133) Fatal exception in thread > Thread[CompactionExecutor:295,1,main] > java.io.IOError: java.io.IOException: dataSize of 14293651161088 > starting at 5541742 would be larger than file > /var/lib/cassandra/data/MSA/modseq-hb-419-Data.db length 10486511 > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:154) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:86) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:70) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:177) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:142) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:134) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:37) > at > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:147) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:124) > at > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) > at > org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:172) > at > org.apache.cassandra.db.compaction.LeveledCompactionTask.execute(LeveledCompactionTask.java:57) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:134) > at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionManager.java:114) > at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.IOException: dataSize of 14293651161088 starting at > 5541742 would be larger than file > /var/lib/cassandra/data/MSA/modseq-hb-419-Data.db length 10486511 > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:115) > ... 23 more > INFO [COMMIT-LOG-WRITER] 2011-12-09 04:36:52,668 > CommitLogSegment.java (line 60) Creating new commitlog segment > /var/lib/cassandra/commitlog/CommitLog-1323405412668.log
