I'm also facing problems regarding corrupt sstables and also couldn't run sstablescrub successfully.
I restarted my nodes with disk failure policy "best_effort", then I run the "nodetool scrub <keyspace>" Once done I removed the corrupt tables manually and started repair On Thu, Oct 1, 2015 at 7:27 PM, John Anderson <son...@gmail.com> wrote: > I have a 25 node cluster and we lost power on one of the racks last night > and now 6 of our nodes will not start up and we are getting the following > error: > > INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is > enabled to receive remote connections on port: 7199 > INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - Initializing > key cache with capacity of 100 MBs. > INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - Initializing > row cache with capacity of 0 MBs > INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - Initializing > counter cache with capacity of 50 MBs > INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling > counter cache save to every 7200 seconds (going to save all keys). > INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 - > Initializing system.sstable_activity > INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 SSTableReader.java:478 > - Opening > /mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45 > (805 bytes) > ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 - > Exiting forcefully due to file system exception on startup, disk failure > policy "stop" > org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_60] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_60] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_60] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_60] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] > Caused by: java.io.EOFException: null > at > java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) > ~[na:1.8.0_60] > at java.io.DataInputStream.readUTF(DataInputStream.java:589) > ~[na:1.8.0_60] > at java.io.DataInputStream.readUTF(DataInputStream.java:564) > ~[na:1.8.0_60] > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) > ~[apache-cassandra-2.1.9.jar:2.1.9] > ... 14 common frames omitted > > > I found some people recommending scrubbing the sstable so I attempted that > and got the following error: > > bin/sstablescrub system sstable_activity -v > > > ERROR 17:26:03 Exiting forcefully due to file system exception on startup, > disk failure policy "stop" > org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534) > ~[apache-cassandra-2.1.9.jar:2.1.9] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_60] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_60] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_60] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_60] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] > Caused by: java.io.EOFException: null > at > java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) > ~[na:1.8.0_60] > at java.io.DataInputStream.readUTF(DataInputStream.java:589) > ~[na:1.8.0_60] > at java.io.DataInputStream.readUTF(DataInputStream.java:564) > ~[na:1.8.0_60] > at > org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) > ~[apache-cassandra-2.1.9.jar:2.1.9] > ... 14 common frames omitted > > > Is there a fix for this? >
