If you don't need LCS, it will be safer to move back to STCS.
If you need LCS, you can manually demote the sstables by editing the
manifest file.
I'm not a man in datastax, and it is a workaround I've just found. Please
test it well before
applying it on your production service.
Background:
You have leveled manifest file for each column family, such like CF1.json
for
a column family "CF1" in data directory. The file maintains sstable
"Level"s.
Before 1.0.9, "cleanup" command has an issue to promote all sstables to
next level. So if you have 10 sstable in L1, those will be compacted and
prompted to L2 after cleanup.
You have finit depth of level(called "generations" in the manifest). If you
are
using default sstable_size_in_mb: 5, the depth will be 8, starting from 0.
( max level is 7 )
If you run "cleanup" repeatedly on a CF for a node, some of your sstables
will be
prompted to the highest level 7.
If you run "cleanup" when you have sstables in L7, cassandra try to promote
them
to L8 which is not exist, and
throw java.lang.ArrayIndexOutOfBoundsException.
Work around:
1. Stop cassandra
2. Remove the manifest files (CF1.json in the example),
3. Start cassandra
Then cassandra add all sstables to L0 and recreate the manifest files.
4. 5 minutes after starting, background compaction will start automatically.
It will compact & promote all sstables from L0.
It will need long time to complete depends on data you have on the node.
hope this help.
maki
On 2012/03/15, at 1:51, Thomas van Neerijnen <t...@bossastudios.com> wrote:
Thanks, good to know there's a fix on the way.
In the mean time is there any way to work around this? Or perhaps should I
move back to SizeTiered until 1.0.9 is released?
On Wed, Mar 14, 2012 at 3:00 PM, Maki Watanabe <watanabe.m...@gmail.com>wrote:
> Fixed in 1.0.9, 1.1.0
> https://issues.apache.org/jira/browse/CASSANDRA-3989
>
> You should better to avoid to use cleanup/scrub/upgradesstable if you
> can on 1.0.7 though
> it will not corrupt sstables.
>
> 2012/3/14 Thomas van Neerijnen <t...@bossastudios.com>:
> > Hi all
> >
> > I am trying to run a cleanup on a column family and am getting the
> following
> > error returned after about 15 seconds. A cleanup on a slightly smaller
> > column family completes in about 21 minutes. This is on the Apache
> packaged
> > version of Cassandra on Ubuntu 11.10, version 1.0.7.
> >
> > ~# nodetool -h localhost cleanup Player PlayerDetail
> > Error occured during cleanup
> > java.util.concurrent.ExecutionException:
> > java.lang.ArrayIndexOutOfBoundsException: 8
> > at
> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> > at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:203)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager.performCleanup(CompactionManager.java:237)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:984)
> > at
> >
> org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1635)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> > at
> >
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> > at
> >
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> > at
> > com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> > at
> > com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> > at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> > at
> > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> > at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
> > at sun.rmi.transport.Transport$1.run(Transport.java:159)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> > at
> > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.lang.ArrayIndexOutOfBoundsException: 8
> > at
> >
> org.apache.cassandra.db.compaction.LeveledManifest.add(LeveledManifest.java:298)
> > at
> >
> org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:186)
> > at
> >
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:141)
> > at
> >
> org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:494)
> > at
> >
> org.apache.cassandra.db.DataTracker.replaceCompactedSSTables(DataTracker.java:234)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.replaceCompactedSSTables(ColumnFamilyStore.java:1006)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:791)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:63)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager$5.perform(CompactionManager.java:241)
> > at
> >
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:182)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> > ... 3 more
> >
>
