David,
There was an issue similar to this reported during the 0.7 development
https://issues.apache.org/jira/browse/CASSANDRA-1744 which was marked as
duplicate with https://issues.apache.org/jira/browse/CASSANDRA-1743 It looks
like the fix for 1743 did not make it into the 0.7 release, but is in the 0.7
branch. I think your's is a bit different but it might pay to test the fix.
I cannot find hudson builds for the 0.7 branch (only trunk builds
https://hudson.apache.org/hudson/job/Cassandra/), are you able to grab the 0.7
branch from svn or git and build it yourself and test it ?
The other possibility is a change in the partitioner
http://www.mail-archive.com/user@cassandra.apache.org/msg02587.html
Cheers
Aaron
On 19 Jan 2011, at 20:10, David King wrote:
> I'm upgrading an 0.6 cluster to 0.7 in a testing environment. In cleaning up
> one of the nodes I get the exception below. Googling around seems to reveal
> people having trouble with it caused by too-small heap sizes but that doesn't
> look to be what's going on here. Am I missing something obvious?
>
> $ time ./cassandra-0.7/bin/nodetool -h cassa7test01 cleanup
> Error occured while cleaning up keyspace keyspace
> java.util.concurrent.ExecutionException: java.io.IOException: Keys must be
> written in ascending order.
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> at
> org.apache.cassandra.db.CompactionManager.performCleanup(CompactionManager.java:180)
> at
> org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:909)
> at
> org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1127)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1449)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1284)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1382)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.IOException: Keys must be written in ascending order.
> at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:107)
> at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:124)
> at
> org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:411)
> at
> org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:54)
> at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:171)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> ... 3 more
>
> real 14m27.895s
> user 0m0.670s
> sys 0m0.200s
>
>
