Attaching the stack dump captured from the last OOM.
Kunal
On 10 July 2015 at 13:32, Kunal Gangakhedkar <kgangakhed...@gmail.com>
wrote:
> Forgot to mention: the data size is not that big - it's barely 10GB in all.
>
> Kunal
>
> On 10 July 2015 at 13:29, Kunal Gangakhedkar <kgangakhed...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have a 2 node setup on Azure (east us region) running Ubuntu server
>> 14.04LTS.
>> Both nodes have 8GB RAM.
>>
>> One of the nodes (seed node) died with OOM - so, I am trying to add a
>> replacement node with same configuration.
>>
>> The problem is this new node also keeps dying with OOM - I've restarted
>> the cassandra service like 8-10 times hoping that it would finish the
>> replication. But it didn't help.
>>
>> The one node that is still up is happily chugging along.
>> All nodes have similar configuration - with libjna installed.
>>
>> Cassandra is installed from datastax's debian repo - pkg: dsc21 version
>> 2.1.7.
>> I started off with the default configuration - i.e. the default
>> cassandra-env.sh - which calculates the heap size automatically (1/4 * RAM
>> = 2GB)
>>
>> But, that didn't help. So, I then tried to increase the heap to 4GB
>> manually and restarted. It still keeps crashing.
>>
>> Any clue as to why it's happening?
>>
>> Thanks,
>> Kunal
>>
>
>
ERROR [SharedPool-Worker-6] 2015-07-10 05:12:16,862
JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[na:1.8.0_45]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_45]
at
org.apache.cassandra.utils.memory.SlabAllocator.getRegion(SlabAllocator.java:137)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:97)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.memory.ContextAllocator.allocate(ContextAllocator.java:57)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.memory.ContextAllocator.clone(ContextAllocator.java:47)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.memory.MemtableBufferAllocator.clone(MemtableBufferAllocator.java:61)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Memtable.put(Memtable.java:192)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1212)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.index.AbstractSimplePerColumnSecondaryIndex.insert(AbstractSimplePerColumnSecondaryIndex.java:131)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.index.SecondaryIndexManager$StandardUpdater.insert(SecondaryIndexManager.java:791)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.AtomicBTreeColumns$ColumnUpdater.apply(AtomicBTreeColumns.java:444)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.AtomicBTreeColumns$ColumnUpdater.apply(AtomicBTreeColumns.java:418)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.utils.btree.BTree.build(BTree.java:116)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.utils.btree.BTree.update(BTree.java:177)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:225)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Memtable.put(Memtable.java:210)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1212)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:389)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:352)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_45]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.7.jar:2.1.7]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
ERROR [CompactionExecutor:3] 2015-07-10 05:12:16,862 CassandraDaemon.java:223 -
Exception in thread Thread[CompactionExecutor:3,1,main]
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayDeque.doubleCapacity(ArrayDeque.java:157)
~[na:1.8.0_45]
at java.util.ArrayDeque.addLast(ArrayDeque.java:247) ~[na:1.8.0_45]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.addColumn(IndexedSliceReader.java:175)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:426)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:350)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:142)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:44)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:89)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:48)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:105)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1927)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1785)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:346)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.SliceQueryPager.queryNextPage(SliceQueryPager.java:90)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:85)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.SliceQueryPager.fetchPage(SliceQueryPager.java:38)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:155)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:144)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:416)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.compaction.CompactionManager$10.run(CompactionManager.java:1144)
~[apache-cassandra-2.1.7.jar:2.1.7]
ERROR [StreamReceiveTask:2] 2015-07-10 05:12:16,862 CassandraDaemon.java:223 -
Exception in thread Thread[StreamReceiveTask:2,5,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:402)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.index.SecondaryIndexManager.maybeBuildSecondaryIndexes(SecondaryIndexManager.java:168)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:136)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[na:1.8.0_45]
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
~[na:1.8.0_45]
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:398)
~[apache-cassandra-2.1.7.jar:2.1.7]
... 7 common frames omitted
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayDeque.doubleCapacity(ArrayDeque.java:157)
~[na:1.8.0_45]
at java.util.ArrayDeque.addLast(ArrayDeque.java:247) ~[na:1.8.0_45]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.addColumn(IndexedSliceReader.java:175)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:426)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:350)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:142)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:44)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:89)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:48)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:105)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1927)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1785)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:346)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.SliceQueryPager.queryNextPage(SliceQueryPager.java:90)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:85)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.SliceQueryPager.fetchPage(SliceQueryPager.java:38)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:155)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:144)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:416)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.compaction.CompactionManager$10.run(CompactionManager.java:1144)
~[apache-cassandra-2.1.7.jar:2.1.7]
ERROR [CompactionExecutor:3] 2015-07-10 05:12:16,864
JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayDeque.doubleCapacity(ArrayDeque.java:157)
~[na:1.8.0_45]
at java.util.ArrayDeque.addLast(ArrayDeque.java:247) ~[na:1.8.0_45]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.addColumn(IndexedSliceReader.java:175)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:426)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:350)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:142)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:44)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:89)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:48)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:105)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1927)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1785)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:346)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.SliceQueryPager.queryNextPage(SliceQueryPager.java:90)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:85)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.SliceQueryPager.fetchPage(SliceQueryPager.java:38)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:155)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:144)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:416)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.compaction.CompactionManager$10.run(CompactionManager.java:1144)
~[apache-cassandra-2.1.7.jar:2.1.7]
ERROR [StreamReceiveTask:2] 2015-07-10 05:12:16,866
JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting
forcefully due to:
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayDeque.doubleCapacity(ArrayDeque.java:157)
~[na:1.8.0_45]
at java.util.ArrayDeque.addLast(ArrayDeque.java:247) ~[na:1.8.0_45]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.addColumn(IndexedSliceReader.java:175)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.getNextBlock(IndexedSliceReader.java:426)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.fetchMoreData(IndexedSliceReader.java:350)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:142)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.computeNext(IndexedSliceReader.java:44)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.jar:na]
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:89)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:48)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:105)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1927)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1785)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:346)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.SliceQueryPager.queryNextPage(SliceQueryPager.java:90)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.AbstractQueryPager.fetchPage(AbstractQueryPager.java:85)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.SliceQueryPager.fetchPage(SliceQueryPager.java:38)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:155)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.pager.QueryPagers$1.next(QueryPagers.java:144)
~[apache-cassandra-2.1.7.jar:2.1.7]
at org.apache.cassandra.db.Keyspace.indexRow(Keyspace.java:416)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.index.SecondaryIndexBuilder.build(SecondaryIndexBuilder.java:62)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.compaction.CompactionManager$10.run(CompactionManager.java:1144)
~[apache-cassandra-2.1.7.jar:2.1.7]
ERROR [CompactionExecutor:3] 2015-07-10 05:12:16,885 CassandraDaemon.java:223 -
Exception in thread Thread[CompactionExecutor:3,1,main]
java.lang.IllegalStateException: Shutdown in progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
~[na:1.8.0_45]
at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.8.0_45]
at
org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:747)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:95)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:64)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.service.CassandraDaemon$1.uncaughtException(CassandraDaemon.java:227)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:229)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
org.apache.cassandra.db.compaction.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:1294)
~[apache-cassandra-2.1.7.jar:2.1.7]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
