Have you dropped and recreated a keyspace with the same name recently?
On Tue, Sep 10, 2013 at 8:40 AM, Keith Freeman <8fo...@gmail.com> wrote: > While running a heavy insert load, one of my nodes started throwing this > exception when trying a compaction: > > INFO [CompactionExecutor:23] 2013-09-09 16:08:07,528 CompactionTask.java > (line 105) Compacting [SSTableReader(p > ath='/var/lib/cassandra/data/**smdb/tracedata/smdb-tracedata-**ic-6-Data.db'), > SSTableReader(path='/var/lib/**cassandr > a/data/smdb/tracedata/smdb-**tracedata-ic-5-Data.db'), > SSTableReader(path='/var/lib/**cassandra/data/smdb/tracedata/ > smdb-tracedata-ic-1-Data.db'), SSTableReader(path='/var/lib/** > cassandra/data/smdb/tracedata/**smdb-tracedata-ic-4-Da > ta.db'), SSTableReader(path='/var/lib/**cassandra/data/smdb/tracedata/** > smdb-tracedata-ic-2-Data.db')] > ERROR [CompactionExecutor:23] 2013-09-09 16:08:07,611 CassandraDaemon.java > (line 192) Exception in thread Thread > [CompactionExecutor:23,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /var/lib/cassandra/data/smdb/**tracedata/smdb-tracedata > -ic-5-Data.db (No such file or directory) > at org.apache.cassandra.io.util.**ThrottledReader.open(** > ThrottledReader.java:53) > at org.apache.cassandra.io.**sstable.SSTableReader.** > openDataReader(SSTableReader.**java:1194) > at org.apache.cassandra.io.**sstable.SSTableScanner.<init>(** > SSTableScanner.java:54) > at org.apache.cassandra.io.**sstable.SSTableReader.** > getDirectScanner(**SSTableReader.java:1014) > at org.apache.cassandra.io.**sstable.SSTableReader.** > getDirectScanner(**SSTableReader.java:1026) > at org.apache.cassandra.db.**compaction.** > AbstractCompactionStrategy.**getScanners(**AbstractCompactionStrategy. > java:157) > at org.apache.cassandra.db.**compaction.** > AbstractCompactionStrategy.**getScanners(**AbstractCompactionStrategy.** > java:163) > at org.apache.cassandra.db.**compaction.CompactionTask.** > runWith(CompactionTask.java:**117) > at org.apache.cassandra.io.util.**DiskAwareRunnable.runMayThrow(** > DiskAwareRunnable.java:48) > at org.apache.cassandra.utils.**WrappedRunnable.run(** > WrappedRunnable.java:28) > at org.apache.cassandra.db.**compaction.CompactionTask.** > executeInternal(**CompactionTask.java:58) > at org.apache.cassandra.db.**compaction.**AbstractCompactionTask.* > *execute(**AbstractCompactionTask.java:**60) > at org.apache.cassandra.db.**compaction.CompactionManager$** > BackgroundCompactionTask.run(**CompactionManager.java:211) > at java.util.concurrent.**Executors$RunnableAdapter.** > call(Executors.java:439) > at java.util.concurrent.**FutureTask$Sync.innerRun(** > FutureTask.java:303) > at java.util.concurrent.**FutureTask.run(FutureTask.**java:138) > at java.util.concurrent.**ThreadPoolExecutor$Worker.** > runTask(ThreadPoolExecutor.**java:895) > at java.util.concurrent.**ThreadPoolExecutor$Worker.run(** > ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.**java:662) > Caused by: java.io.FileNotFoundException: /var/lib/cassandra/data/smdb/** > tracedata/smdb-tracedata-ic-5-**Data.db (No such file or directory) > at java.io.RandomAccessFile.open(**Native Method) > at java.io.RandomAccessFile.<**init>(RandomAccessFile.java:**216) > at org.apache.cassandra.io.util.**RandomAccessReader.<init>(** > RandomAccessReader.java:67) > at org.apache.cassandra.io.util.**ThrottledReader.<init>(** > ThrottledReader.java:35) > at org.apache.cassandra.io.util.**ThrottledReader.open(** > ThrottledReader.java:49) > ... 18 more > > This shows up many times in the log. I figured running a repair on the > node might fix it, but the repair ran for over an hour (the node only has > about 3G of data), so I figured it was hung. I tried restarting the > repair, but each time it starts the node logs that same exception > immediately: > > INFO [AntiEntropySessions:5] 2013-09-10 09:36:35,526 > AntiEntropyService.java (line 651) [repair #c6ab9c00-1a2e- > 11e3-b0e5-05d1729cecff] new session: will sync /192.168.27.73, / > 192.168.27.75 on range (4925454539472655923,4991 > 066214171147775] for smdb.[tracedata, processors] > INFO [AntiEntropySessions:5] 2013-09-10 09:36:35,526 > AntiEntropyService.java (line 857) [repair #c6ab9c00-1a2e- > 11e3-b0e5-05d1729cecff] requesting merkle trees for tracedata (to [/ > 192.168.27.75, /192.168.27.73]) > ERROR [ValidationExecutor:2] 2013-09-10 09:36:35,535 CassandraDaemon.java > (line 192) Exception in thread Thread[ > ValidationExecutor:2,1,main] > java.lang.RuntimeException: java.io.FileNotFoundException: > /var/lib/cassandra/data/smdb/**tracedata/smdb-tracedata > -ic-5-Data.db (No such file or directory) > at org.apache.cassandra.io.util.**ThrottledReader.open(** > ThrottledReader.java:53) > at org.apache.cassandra.io.**sstable.SSTableReader.** > openDataReader(SSTableReader.**java:1194) > ... > > How can I fix this node? > > > >
