On Friday 18 of March 2011, Jonathan Ellis wrote: > Upgrade to 0.7.4 and run nodetool scrub (then watch the log).
Unfortunately nodetool scrub fails with: WARN [CompactionExecutor:1] 2011-03-19 00:34:53,511 CompactionManager.java (line 607) Non-fatal error reading row (stacktrace follows) java.io.IOError: java.io.IOException: Impossible row size 3458764513820542976 at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:589) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Impossible row size 3458764513820542976 ... 8 more INFO [CompactionExecutor:1] 2011-03-19 00:34:53,512 CompactionManager.java (line 613) Retrying from row index; data is 10296 bytes starting at 2648761 ERROR [CompactionExecutor:1] 2011-03-19 00:34:53,512 AbstractCassandraDaemon.java (line 112) Fatal exception in thread Thread[CompactionExecutor:1,1,main] java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:49) at org.apache.cassandra.utils.BloomFilterSerializer.deserialize(BloomFilterSerializer.java:30) at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:117) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:87) at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:618) at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56) at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) > If it > reports bad rows then run repair post-scrub. You mean: nodetool repair ? Thanks for help, regards, > > On Fri, Mar 18, 2011 at 6:44 AM, Mateusz Korniak > > <mateusz-li...@ant.gliwice.pl> wrote: > > Hi ! > > I have run with Cassandra 0.7.2 out of disc space and after moving to > > bigger partition I experience compaction failures[1]. > > > > 1) I suspect one of SSTables is broken. If I am right how can I find > > which one exactly ? > > > > 2) Knowing which one is broken is it safe to stop Cassandra, remove > > -Data.db -Filter.db -Index.db -Statistics.db of broken SSTable and > > restart ? Sure I will loose data but it's one of RF=3 nodes so it's not > > big problem. > > > > Any suggestions, hints ? > > Thanks in advance, regards, > > > > [1] > > INFO [CompactionExecutor:1] 2011-03-18 12:27:17,103 > > CompactionManager.java (line 458) Compacted to > > /var/lib/cassandra/data/foo/bar-tmp-f-707-Data.db. 42,276,185 to > > 42,103,628 (~99% of original) bytes for 33,869 keys. Time: 25,924ms. > > INFO [CompactionExecutor:1] 2011-03-18 12:27:17,133 > > CompactionManager.java (line 373) Compacting > > [org.apache.cassandra.io.sstable.SSTableReader(path= > > '/var/lib/cassandra/data/foo/bar-f-238-Data.db'), > > org.apache.cassandra.io.sstable.SSTableReader( > > path='/var/lib/cassandra/data/foo/bar-f-243-Data.db'), > > org.apache.cassandra.io.sstable.SSTableReader( > > path='/var/lib/cassandra/data/foo/bar-f-361-Data.db'), > > org.apache.cassandra.io.sstable.SSTableReader( > > path='/var/lib/cassandra/data/foo/bar-f-635-Data.db'), > > org.apache.cassandra.io.sstable.SSTableReader( > > path='/var/lib/cassandra/data/foo/bar-f-643-Data.db'), > > org.apache.cassandra.io.sstable.SSTableReader( > > path='/var/lib/cassandra/data/foo/bar-f-680-Data.db'), > > org.apache.cassandra.io.sstable.SSTableReader( > > path='/var/lib/cassandra/data/foo/bar-f-684-Data.db'), > > org.apache.cassandra.io.sstable.SSTableReader( > > path='/var/lib/cassandra/data/foo/bar-f-685-Data.db'), > > org.apache.cassandra.io.sstable.SSTableReader( > > path='/var/lib/cassandra/data/foo/bar-f-687-Data.db')] > > ERROR [CompactionExecutor:1] 2011-03-18 12:27:18,241 > > AbstractCassandraDaemon.java (line 114) Fatal exception in thread > > Thread[CompactionExecutor:1,1,main] > > java.io.IOError: java.io.EOFException > > at > > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIde > > ntityIterator.java:78) at > > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(S > > STableScanner.java:179) at > > org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(S > > STableScanner.java:144) at > > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:1 > > 36) at > > org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:3 > > 9) at > > org.apache.commons.collections.iterators.CollatingIterator.set(CollatingI > > terator.java:284) at > > org.apache.commons.collections.iterators.CollatingIterator.least(Collatin > > gIterator.java:326) at > > org.apache.commons.collections.iterators.CollatingIterator.next(Collating > > Iterator.java:230) at > > org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator. > > java:68) at > > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractItera > > tor.java:136) at > > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java: > > 131) at > > org.apache.commons.collections.iterators.FilterIterator.setNextObject(Fil > > terIterator.java:183) at > > org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIte > > rator.java:94) at > > org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager. > > java:427) at > > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:1 > > 23) at > > org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:9 > > 3) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > > at java.util.concurrent.FutureTask.run(FutureTask.java:138) at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor > > .java:886) at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav > > a:908) at java.lang.Thread.run(Thread.java:662) > > Caused by: java.io.EOFException > > at > > org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65 > > ) at > > org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIde > > ntityIterator.java:70) ... 20 more > > > > > > -- > > Mateusz Korniak -- Mateusz Korniak