I've been using nodetool move to rebalance my cluster. Most of the moves take
under an hour, or a few hours at most. The current move has taken 4+ days so
I'm afraid it will never complete. What's the best way to cancel it and try
again?
I'm running a cluster of 12 nodes at AWS. Each node runs Cassandra 1.2.5 on an
m1.xlarge EC2 instance, and they are spread across 3 availability zones within
a single region.
I've seen some of these errors in the log. I'm not sure if it's related or not:
ERROR [CompactionExecutor:4092] 2013-09-10 01:31:49,783 CassandraDaemon.java
(line 175) Exception in thread Thread[CompactionExecutor:4092,1,main]
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
at
com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
at
com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
at
org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:128)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:219)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
at
org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:98)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:134)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:211)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Here's the status of the cluster as reported by nodetool ring, showing the one
node in "Moving" state:
Datacenter: us-east
==========
Address Rack Status State Load Owns
Token
127605887595351923798765477786913079290
10.xxx.xxx.xxx 1c Up Normal 224.53 GB 25.00%
0
10.xxx.xxx.xxx 1d Up Moving 297.46 GB 2.44%
4150051970709140963435425752946440221
10.xxx.xxx.xxx 1d Up Normal 107.75 GB 5.89%
14178431955039102644307275309657008810
10.xxx.xxx.xxx 1e Up Normal 82.75 GB 8.33%
28356863910078205288614550619314017620
10.xxx.xxx.xxx 1e Up Normal 173.83 GB 2.99%
33451586107772559423309548485325625873
10.xxx.xxx.xxx 1c Up Normal 64.4 GB 5.34%
42535295865117307932921825928971026430
10.xxx.xxx.xxx 1d Up Normal 56.46 GB 8.33%
56713727820156410577229101238628035240
10.xxx.xxx.xxx 1e Up Normal 57.88 GB 8.33%
70892159775195513221536376548285044050
10.xxx.xxx.xxx 1c Up Normal 95.18 GB 8.33%
85070591730234615865843651857942052860
10.xxx.xxx.xxx 1d Up Normal 169.37 GB 8.33%
99249023685273718510150927167599061670
10.xxx.xxx.xxx 1e Up Normal 109.99 GB 8.33%
113427455640312821154458202477256070480
10.xxx.xxx.xxx 1c Up Normal 98.64 GB 8.33%
127605887595351923798765477786913079290
