This is probably due to corrupt data or a cassandra upgrade where you didn't ran upgradesstables
I'd then suggest scrubbing the column family (or upgrading it). Hope it helps. Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso> On 31 March 2016 at 12:10, <m...@vrischmann.me> wrote: > Hi all, > > Recently we tried to repair one of our biggest table, and we keep getting > hit by errors related to hard link. Here's a stacktrace: > > ERROR [RepairJobTask:4] 2016-03-31 05:47:27,268 RepairJob.java:145 - Error > occurred during snapshot phase > java.lang.RuntimeException: Could not create snapshot at /10.51.0.7 > at > org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:48) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [na:1.7.0_80] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_80] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80] > ERROR [AntiEntropyStage:39] 2016-03-31 05:47:27,268 > CassandraDaemon.java:223 - Exception in thread > Thread[AntiEntropyStage:39,5,main] > java.lang.RuntimeException: java.lang.RuntimeException: Tried to hard link > to file that does not exist > /data/db/ks/table-a24af0002ed511e5b983ade99871dd76/ks-table-ka-50582-Statistics.db > at > org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:141) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_80] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0_80] > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80] > Caused by: java.lang.RuntimeException: Tried to hard link to file that > does not exist > /data/db/ks/table-a24af0002ed511e5b983ade99871dd76/ks-table-ka-50582-Statistics.db > at > org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:90) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1799) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:2237) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:2319) > ~[apache-cassandra-2.1.5.jar:2.1.5] > at > org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:82) > ~[apache-cassandra-2.1.5.jar:2.1.5] > ... 4 common frames omitted > > I tried Googling for that particular error and I did not find a definitive > answer, instead what seems to be recommended is to simply restart the node. > However, we're getting this error at least once a day and sometimes on > multiple nodes (we have 7 nodes currently), so it's getting tedious to > restart cassandra every time. > > I saw the issue https://issues.apache.org/jira/browse/CASSANDRA-6433 and > it suggests it's due to a drop of a keyspace, but we didn't do any drop. So > I'm not sure that issue really applies, although the error is related. > > This issue https://issues.apache.org/jira/browse/CASSANDRA-6716 reports > the same exception but we didn't do any scrubbing, so I'm not sure it > applies either. > > We're running cassandra 2.1.5 by the way. I don't know if upgrading will > fix the problems, because I didn't really see anything related to this > looking in the changelogs. > > I'm wondering if getting these exceptions will somehow "block" the repair, > because it seems the repair is super slow right now (we're talking days > repairing). >
