Dear All, Recently one of node in our cluster has high cpu load ~100%. It seems to me there is a infinite loop in SliceQueryFilter.
The below log is repeated in 5000ms (range_request_timeout_in_ms). TRACE [SharedPool-Worker-11] 2015-12-04 19:25:33,418 SliceQueryFilter.java:269 - collecting 1 of 2147483647: images:b608719e728d11e5812b57f4c5416142:false:62@1444838867382000 Our version is 2.19. Here is the bt: org.apache.cassandra.db.composites.AbstractComposite.isEmpty(AbstractComposite.java:30) org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:76) org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83) org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37) com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173) org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156) org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146) org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125) org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:264) org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:108) org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82) org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69) org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314) org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2033) org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1876) org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:357) org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85) org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47) org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) java.lang.Thread.run(Thread.java:745) Appreciate if anyone could help. Thanks in advance, Xihui
