Greetings, We recently upgraded from 1.0.8 to 1.1.0. Everything has been running fine with the exception of snapshots. When attempting to snapshot any of the nodes in our six node cluster we are seeing the following error.
[root@cassandra-n6 blotter]# /opt/apache-cassandra-1.1.0/bin/nodetool -h 10.20.50.58 snapshot Requested snapshot for: all keyspaces Exception in thread "main" java.io.IOError: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json (errno 17) at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454) at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483) at org.apache.cassandra.db.Table.snapshot(Table.java:205) at org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:1793) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Unable to create hard link from /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json (errno 17) at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163) at org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343) at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450) ... 33 more However, an LS shows that both of these JSON files exist on the filesystem (although slightly different sizes). [root@cassandra-n6 blotter]# ls -al /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json -rw-r--r-- 1 root root 38786 May 15 20:51 /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json [root@cassandra-n6 blotter]# ls -al /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json -rw-r--r-- 1 root root 38778 May 15 20:50 /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json We are using Leveled Compaction on the twitter_users CF with I assume is creating the JSON files. [root@cassandra-n6 blotter]# ls -al /var/lib/cassandra/data/blotter/twitter_users/*.json -rw-r--r-- 1 root root 38779 May 15 20:51 /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json -rw-r--r-- 1 root root 38779 May 15 20:51 /var/lib/cassandra/data/blotter/twitter_users/twitter_users-old.json -rw-r--r-- 1 root root 1040 May 15 20:51 /var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_user_attributes_screenname_idx.json -rw-r--r-- 1 root root 1046 May 15 20:50 /var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_user_attributes_screenname_idx-old.json The other column families which are not using Leveled Compaction seem to have their snapshots created successfully. Any ideas other than turning off Leveled Compaction? Thanks, Brian
