I got cassandra-reaper up and running the other day on our 5 node 2.2.8
cluster. Now every once in awhile a node will get the below exception and the
node shuts down its cql port. Is this a bug? something I am doing wrong? Are
there any workarounds?
-Jacob Willoughby
ERROR [AntiEntropyStage:1] 2017-01-25 12:55:25,609 CassandraDaemon.java:195 -
Exception in thread Thread[AntiEntropyStage:1,5,main]
org.apache.cassandra.io.FSWriteError: java.nio.file.NoSuchFileException:
/data/cassandra/data/Objects/Objects-11baea008ccf11e6be00d321fe82b612/snapshots/8a2da7f0-e2fd-11e6-a9da-e96a8b206d4c/lb-127076-big-CompressionInfo.db
-> /data/cassandra/data/O
bjects/Objects-11baea008ccf11e6be00d321fe82b612/lb-127076-big-CompressionInfo.db
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:92)
~[apache-cassandra-2.2.8.jar:2.2.8]
at
org.apache.cassandra.io.sstable.format.SSTableReader.createLinks(SSTableReader.java:1777)
~[apache-cassandra-2.2.8.jar:2.2.8]
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:2432)
~[apache-cassandra-2.2.8.jar:2.2.8]
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2557)
~[apache-cassandra-2.2.8.jar:2.2.8]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:110)
~[apache-cassandra-2.2.8.jar:2.2.8]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-2.2.8.jar:2.2.8]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_121]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: java.nio.file.NoSuchFileException:
/data/cassandra/data/Objects/Objects-11baea008ccf11e6be00d321fe82b612/snapshots/8a2da7f0-e2fd-11e6-a9da-e96a8b206d4c/lb-127076-big-CompressionInfo.db
-> /data/cassandra/data/Objects/Objects-11baea008ccf
11e6be00d321fe82b612/lb-127076-big-CompressionInfo.db
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
~[na:1.8.0_121]
at
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
~[na:1.8.0_121]
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
~[na:1.8.0_121]
at java.nio.file.Files.createLink(Files.java:1086) ~[na:1.8.0_121]
at
org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:88)
~[apache-cassandra-2.2.8.jar:2.2.8]
... 10 common frames omitted
ERROR [AntiEntropyStage:1] 2017-01-25 12:55:25,611 StorageService.java:388 -
Stopping gossiper
WARN [AntiEntropyStage:1] 2017-01-25 12:55:25,611 StorageService.java:284 -
Stopping gossip by operator request
INFO [AntiEntropyStage:1] 2017-01-25 12:55:25,611 Gossiper.java:1451 -
Announcing shutdown
INFO [AntiEntropyStage:1] 2017-01-25 12:55:25,612 StorageService.java:1918 -
Node /10.0.3.229 state jump to shutdown
