Upgrade to 0.7.4 and run nodetool scrub (then watch the log). If it reports bad rows then run repair post-scrub.
On Fri, Mar 18, 2011 at 6:44 AM, Mateusz Korniak <mateusz-li...@ant.gliwice.pl> wrote: > Hi ! > I have run with Cassandra 0.7.2 out of disc space and after moving to bigger > partition I experience compaction failures[1]. > > 1) I suspect one of SSTables is broken. If I am right how can I find which one > exactly ? > > 2) Knowing which one is broken is it safe to stop Cassandra, remove > -Data.db -Filter.db -Index.db -Statistics.db of broken SSTable and restart ? > Sure I will loose data but it's one of RF=3 nodes so it's not big problem. > > Any suggestions, hints ? > Thanks in advance, regards, > > [1] > INFO [CompactionExecutor:1] 2011-03-18 12:27:17,103 CompactionManager.java > (line 458) Compacted to /var/lib/cassandra/data/foo/bar-tmp-f-707-Data.db. > 42,276,185 to 42,103,628 (~99% of original) bytes for 33,869 keys. Time: > 25,924ms. > INFO [CompactionExecutor:1] 2011-03-18 12:27:17,133 CompactionManager.java > (line 373) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path= > '/var/lib/cassandra/data/foo/bar-f-238-Data.db'), > org.apache.cassandra.io.sstable.SSTableReader( > path='/var/lib/cassandra/data/foo/bar-f-243-Data.db'), > org.apache.cassandra.io.sstable.SSTableReader( > path='/var/lib/cassandra/data/foo/bar-f-361-Data.db'), > org.apache.cassandra.io.sstable.SSTableReader( > path='/var/lib/cassandra/data/foo/bar-f-635-Data.db'), > org.apache.cassandra.io.sstable.SSTableReader( > path='/var/lib/cassandra/data/foo/bar-f-643-Data.db'), > org.apache.cassandra.io.sstable.SSTableReader( > path='/var/lib/cassandra/data/foo/bar-f-680-Data.db'), > org.apache.cassandra.io.sstable.SSTableReader( > path='/var/lib/cassandra/data/foo/bar-f-684-Data.db'), > org.apache.cassandra.io.sstable.SSTableReader( > path='/var/lib/cassandra/data/foo/bar-f-685-Data.db'), > org.apache.cassandra.io.sstable.SSTableReader( > path='/var/lib/cassandra/data/foo/bar-f-687-Data.db')] > ERROR [CompactionExecutor:1] 2011-03-18 12:27:18,241 > AbstractCassandraDaemon.java (line 114) Fatal exception in thread > Thread[CompactionExecutor:1,1,main] > java.io.IOError: java.io.EOFException > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:78) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:179) > at > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:144) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:136) > at > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:39) > at > org.apache.commons.collections.iterators.CollatingIterator.set(CollatingIterator.java:284) > at > org.apache.commons.collections.iterators.CollatingIterator.least(CollatingIterator.java:326) > at > org.apache.commons.collections.iterators.CollatingIterator.next(CollatingIterator.java:230) > at > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:68) > at > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:136) > at > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:131) > at > org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183) > at > org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94) > at > org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:427) > at > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:123) > at > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:93) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.EOFException > at > org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65) > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:70) > ... 20 more > > > -- > Mateusz Korniak > -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
