I have upgraded from 0.7.0 to 0.7.3. I then run nodetool scrub on my keyspace and now see this exception:
Exception in thread "main" java.io.IOError: java.io.IOException: Cannot run program "ln": java.io.IOException: error=24, Too many open files at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1658) at org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:962) at org.apache.cassandra.service.StorageService.scrub(StorageService.java:1256) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.IOException: Cannot run program "ln": java.io.IOException: error=24, Too many open files at java.lang.ProcessBuilder.start(ProcessBuilder.java:475) at org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:181) at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:147) at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:617) at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1651) ... 32 more Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files at java.lang.UNIXProcess.<init>(UNIXProcess.java:164) at java.lang.ProcessImpl.start(ProcessImpl.java:81) at java.lang.ProcessBuilder.start(ProcessBuilder.java:468) ... 36 more On Thu, Mar 10, 2011 at 5:49 PM, aaron morton <aa...@thelastpickle.com>wrote: > What version of cassandra are you using and what is the upgrade history for > the cluster? > Aaron > > On 10/03/2011, at 8:24 PM, Stu King wrote: > > > I am seeing this exception when I am trying to run a cleanup. I want to > decommission the node after the cleanup. > > > > java.util.concurrent.ExecutionException: java.io.IOError: > java.io.EOFException > > at > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) > > at java.util.concurrent.FutureTask.get(FutureTask.java:111) > > at > org.apache.cassandra.db.CompactionManager.performCleanup(CompactionManager.java:180) > > at > org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:909) > > at > org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1127) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:616) > > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111) > > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45) > > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226) > > at > com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138) > > at > com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251) > > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857) > > at > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795) > > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450) > > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90) > > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285) > > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383) > > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:616) > > at > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) > > at sun.rmi.transport.Transport$1.run(Transport.java:177) > > at java.security.AccessController.doPrivileged(Native Method) > > at sun.rmi.transport.Transport.serviceCall(Transport.java:173) > > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) > > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) > > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > Caused by: java.io.IOError: java.io.EOFException > > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:105) > > at > org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:418) > > at > org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:54) > > at > org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:171) > > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > > ... 3 more > > Caused by: java.io.EOFException > > at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416) > > at > org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280) > > at > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76) > > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:364) > > at > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313) > > at > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101) > > ... 8 more > >
