While running a heavy insert load, one of my nodes started throwing this
exception when trying a compaction:
INFO [CompactionExecutor:23] 2013-09-09 16:08:07,528
CompactionTask.java (line 105) Compacting [SSTableReader(p
ath='/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata-ic-6-Data.db'),
SSTableReader(path='/var/lib/cassandr
a/data/smdb/tracedata/smdb-tracedata-ic-5-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/smdb/tracedata/
smdb-tracedata-ic-1-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata-ic-4-Da
ta.db'),
SSTableReader(path='/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata-ic-2-Data.db')]
ERROR [CompactionExecutor:23] 2013-09-09 16:08:07,611
CassandraDaemon.java (line 192) Exception in thread Thread
[CompactionExecutor:23,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException:
/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata
-ic-5-Data.db (No such file or directory)
at
org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1194)
at
org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:54)
at
org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1014)
at
org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1026)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.
java:157)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException:
/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata-ic-5-Data.db (No
such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
at
org.apache.cassandra.io.util.ThrottledReader.<init>(ThrottledReader.java:35)
at
org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:49)
... 18 more
This shows up many times in the log. I figured running a repair on the
node might fix it, but the repair ran for over an hour (the node only
has about 3G of data), so I figured it was hung. I tried restarting the
repair, but each time it starts the node logs that same exception
immediately:
INFO [AntiEntropySessions:5] 2013-09-10 09:36:35,526
AntiEntropyService.java (line 651) [repair #c6ab9c00-1a2e-
11e3-b0e5-05d1729cecff] new session: will sync /192.168.27.73,
/192.168.27.75 on range (4925454539472655923,4991
066214171147775] for smdb.[tracedata, processors]
INFO [AntiEntropySessions:5] 2013-09-10 09:36:35,526
AntiEntropyService.java (line 857) [repair #c6ab9c00-1a2e-
11e3-b0e5-05d1729cecff] requesting merkle trees for tracedata (to
[/192.168.27.75, /192.168.27.73])
ERROR [ValidationExecutor:2] 2013-09-10 09:36:35,535
CassandraDaemon.java (line 192) Exception in thread Thread[
ValidationExecutor:2,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException:
/var/lib/cassandra/data/smdb/tracedata/smdb-tracedata
-ic-5-Data.db (No such file or directory)
at
org.apache.cassandra.io.util.ThrottledReader.open(ThrottledReader.java:53)
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1194)
...
How can I fix this node?
