Hi Aaron,
We are using Ubuntu (AMI Datastax 1.0.9 as I said).
Release: 10.10
Codename: maverick
ERROR [RMI TCP Connection(37732)-10.248.10.94] 2012-06-13 15:00:17,157
CLibrary.java (line 153) Unable to create hard link
com.sun.jna.LastErrorException: errno was 1
at org.apache.cassandra.utils.CLibrary.link(Native Method)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:145)
at
org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:857)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1412)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462)
at org.apache.cassandra.db.Table.snapshot(Table.java:210)
at
org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:1710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
WARN [RMI TCP Connection(37732)-10.248.10.94] 2012-06-13 15:00:17,158
CLibrary.java (line 97) Obsolete version of JNA present; unable to
read errno. Upgrade to JNA 3.2.7 or later
I see that a JNA update seems necessary. Shouldn't this AMI be fully
worrking "out of the box" ?
I am going to search how to upgrade JNA.
Alain
2012/5/31 aaron morton <aa...@thelastpickle.com>:
> CASSANDRA-4230 is a bug in 1.1
>
> I am not aware of issues using snapshot on 1.0.9. But errno 0 is a bit odd.
>
> On the server side there should be a log message at ERROR level that
> contains the string "Unable to create hard link" and the error message. What
> does that say ?
>
> Can you also include the OS version.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 28/05/2012, at 9:27 PM, Alain RODRIGUEZ wrote:
>
> I have the same error with the last Datastax AMI (1.0.9). Is that the same
> bug ?
>
> Requested snapshot for: cassa_teads
> Exception in thread "main" java.io.IOError: java.io.IOException:
> Unable to create hard link from
> /raid0/cassandra/data/cassa_teads/stats_product-hc-233-Index.db to
> /raid0/cassandra/data/cassa_teads/snapshots/20120528/stats_product-hc-233-Index.db
> (errno 0)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1433)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1462)
> at org.apache.cassandra.db.Table.snapshot(Table.java:210)
> at
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:1710)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: Unable to create hard link from
> /raid0/cassandra/data/cassa_teads/stats_product-hc-233-Index.db to
> /raid0/cassandra/data/cassa_teads/snapshots/20120528/stats_product-hc-233-Index.db
> (errno 0)
> at
> org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:158)
> at
> org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:857)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1412)
> ... 32 more
>
> Can we do a snapshot manually (like flushing and after copying all the
> file into the snapshot folder) ?
>
> Alain
>
> 2012/5/19 Jonathan Ellis <jbel...@gmail.com>:
>
> When these bugs are fixed:
>
> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+CASSANDRA+AND+fixVersion+%3D+%221.1.1%22+AND+resolution+%3D+Unresolved+ORDER+BY+due+ASC%2C+priority+DESC%2C+created+ASC&mode=hide
>
>
> On Wed, May 16, 2012 at 6:35 PM, Bryan Fernandez <bfernande...@gmail.com>
> wrote:
>
> Does anyone know when 1.1.1 will be released?
>
>
> Thanks.
>
>
> On Tue, May 15, 2012 at 5:40 PM, Brandon Williams <dri...@gmail.com> wrote:
>
>
> Probably https://issues.apache.org/jira/browse/CASSANDRA-4230
>
>
> On Tue, May 15, 2012 at 4:08 PM, Bryan Fernandez <bfernande...@gmail.com>
>
> wrote:
>
> Greetings,
>
>
> We recently upgraded from 1.0.8 to 1.1.0. Everything has been running
>
> fine
>
> with the exception of snapshots. When attempting to snapshot any of the
>
> nodes in our six node cluster we are seeing the following error.
>
>
> [root@cassandra-n6 blotter]# /opt/apache-cassandra-1.1.0/bin/nodetool -h
>
> 10.20.50.58 snapshot
>
> Requested snapshot for: all keyspaces
>
> Exception in thread "main" java.io.IOError: java.io.IOException: Unable
>
> to
>
> create hard link from
>
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to
>
>
> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
>
> (errno 17)
>
> at
>
>
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
>
> at
>
>
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
>
> at org.apache.cassandra.db.Table.snapshot(Table.java:205)
>
> at
>
>
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:1793)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
>
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
>
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at
>
>
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
>
> at
>
>
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
>
> at
>
>
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
>
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
>
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
>
> at
>
>
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
>
> at
>
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
>
> at
>
>
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
>
> at
>
>
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
>
> at
>
>
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
>
> at
>
>
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
>
> at
>
>
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
>
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> at
>
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:597)
>
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
>
> at sun.rmi.transport.Transport$1.run(Transport.java:159)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>
> at
>
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
>
> at
>
>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
>
> at
>
>
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
>
> at
>
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> at
>
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>
> at java.lang.Thread.run(Thread.java:662)
>
> Caused by: java.io.IOException: Unable to create hard link from
>
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to
>
>
> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
>
> (errno 17)
>
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
>
> at
>
>
> org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
>
> at
>
>
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
>
> ... 33 more
>
>
>
> However, an LS shows that both of these JSON files exist on the
>
> filesystem
>
> (although slightly different sizes).
>
>
> [root@cassandra-n6 blotter]# ls -al
>
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json
>
> -rw-r--r-- 1 root root 38786 May 15 20:51
>
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json
>
>
> [root@cassandra-n6 blotter]# ls -al
>
>
> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
>
> -rw-r--r-- 1 root root 38778 May 15 20:50
>
>
> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
>
>
>
> We are using Leveled Compaction on the twitter_users CF with I assume is
>
> creating the JSON files.
>
>
> [root@cassandra-n6 blotter]# ls -al
>
> /var/lib/cassandra/data/blotter/twitter_users/*.json
>
> -rw-r--r-- 1 root root 38779 May 15 20:51
>
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json
>
> -rw-r--r-- 1 root root 38779 May 15 20:51
>
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users-old.json
>
> -rw-r--r-- 1 root root 1040 May 15 20:51
>
>
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_user_attributes_screenname_idx.json
>
> -rw-r--r-- 1 root root 1046 May 15 20:50
>
>
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_user_attributes_screenname_idx-old.json
>
>
>
> The other column families which are not using Leveled Compaction seem to
>
> have their snapshots created successfully.
>
>
> Any ideas other than turning off Leveled Compaction?
>
>
>
> Thanks,
>
>
> Brian
>
>
>
>
>
>
>
>
>
>
> --
>
> Jonathan Ellis
>
> Project Chair, Apache Cassandra
>
> co-founder of DataStax, the source for professional Cassandra support
>
> http://www.datastax.com
>
>
