May just be going over a lot of data. Does output of 'nodetool cfstats' show large partitions? (partition maximum bytes). "collecting 1 of 2147483647" is suspicious. Are your queries using ALLOW FILTERING or have very high limits? If trying to read 2 billion entries in 1 query you will have memory issues. May want to check with jvmtop/htop to make sure its not GCs using CPU as well. Is there a sane amount of sstables? Providing some more details can help (cfstats, cfhistograms, queries your making, schema)
Chris On Fri, Dec 4, 2015 at 10:43 AM, Xihui He <xihu...@gmail.com> wrote: > Dear All, > > Recently one of node in our cluster has high cpu load ~100%. It seems to > me there is a infinite loop in SliceQueryFilter. > > The below log is repeated in 5000ms (range_request_timeout_in_ms). > TRACE [SharedPool-Worker-11] 2015-12-04 19:25:33,418 > SliceQueryFilter.java:269 - collecting 1 of 2147483647: > images:b608719e728d11e5812b57f4c5416142:false:62@1444838867382000 > > Our version is 2.19. Here is the bt: > > org.apache.cassandra.db.composites.AbstractComposite.isEmpty(AbstractComposite.java:30) > > org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:76) > org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52) > org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46) > > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:83) > > org.apache.cassandra.db.columniterator.SimpleSliceReader.computeNext(SimpleSliceReader.java:37) > > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > > org.apache.cassandra.db.columniterator.SSTableSliceIterator.hasNext(SSTableSliceIterator.java:82) > org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:173) > org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:156) > > org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146) > > org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:125) > > org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:99) > > com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) > > com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) > > org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:264) > > org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:108) > > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82) > > org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69) > > org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314) > > org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) > > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2033) > > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1876) > org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:357) > > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85) > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47) > > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > > org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) > org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > > java.lang.Thread.run(Thread.java:745) > > Appreciate if anyone could help. > > Thanks in advance, > Xihui >
