I have a 25 node cluster and we lost power on one of the racks last night
and now 6 of our nodes will not start up and we are getting the following
error:
INFO [main] 2015-10-01 10:19:22,111 CassandraDaemon.java:122 - JMX is
enabled to receive remote connections on port: 7199
INFO [main] 2015-10-01 10:19:22,124 CacheService.java:111 - Initializing
key cache with capacity of 100 MBs.
INFO [main] 2015-10-01 10:19:22,129 CacheService.java:133 - Initializing
row cache with capacity of 0 MBs
INFO [main] 2015-10-01 10:19:22,133 CacheService.java:150 - Initializing
counter cache with capacity of 50 MBs
INFO [main] 2015-10-01 10:19:22,135 CacheService.java:161 - Scheduling
counter cache save to every 7200 seconds (going to save all keys).
INFO [main] 2015-10-01 10:19:22,211 ColumnFamilyStore.java:363 -
Initializing system.sstable_activity
INFO [SSTableBatchOpen:1] 2015-10-01 10:19:22,639 SSTableReader.java:478 -
Opening
/mnt/cassandra/data/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-45
(805 bytes)
ERROR [SSTableBatchOpen:1] 2015-10-01 10:19:22,657 FileUtils.java:447 -
Exiting forcefully due to file system exception on startup, disk failure
policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: java.io.EOFException: null
at
java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
~[na:1.8.0_60]
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
~[na:1.8.0_60]
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
~[na:1.8.0_60]
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
~[apache-cassandra-2.1.9.jar:2.1.9]
... 14 common frames omitted
I found some people recommending scrubbing the sstable so I attempted that
and got the following error:
bin/sstablescrub system sstable_activity -v
ERROR 17:26:03 Exiting forcefully due to file system exception on startup,
disk failure policy "stop"
org.apache.cassandra.io.sstable.CorruptSSTableException:
java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:752)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:703)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:387)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:534)
~[apache-cassandra-2.1.9.jar:2.1.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_60]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
Caused by: java.io.EOFException: null
at
java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
~[na:1.8.0_60]
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
~[na:1.8.0_60]
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
~[na:1.8.0_60]
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106)
~[apache-cassandra-2.1.9.jar:2.1.9]
... 14 common frames omitted
Is there a fix for this?
