Hi,
We are not exactly sure what is causing this problem. But after compaction
happens (after 1 week ttl). We start getting this exception:
WARN [SharedPool-Worker-1] 2016-09-26 04:07:19,849
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.NullPointerException: null
at
org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_102]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.9.jar:3.0.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
WARN [SharedPool-Worker-2] 2016-09-26 04:07:20,639
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-2,5,main]: {}
java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2470)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_102]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-3.0.9.jar:3.0.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
Caused by: java.lang.NullPointerException: null
at
org.apache.cassandra.db.Slices$ArrayBackedSlices$ComponentOfSlice.isEQ(Slices.java:748)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.Slices$ArrayBackedSlices.toCQLString(Slices.java:659)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.filter.ClusteringIndexSliceFilter.toCQLString(ClusteringIndexSliceFilter.java:150)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.SinglePartitionReadCommand.appendCQLWhereClause(SinglePartitionReadCommand.java:911)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommand.toCQLString(ReadCommand.java:560)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:506)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:76)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797)
~[apache-cassandra-3.0.9.jar:3.0.9]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466)
~[apache-cassandra-3.0.9.jar:3.0.9]
... 5 common frames omitted
We have no idea how to solve this. I am losing sleep over this, please help!
Cassandra: 3.0.9 (also happened in 3.0.8)
Java: Oracle jdk "1.8.0_102"
jemalloc enabled
Regards,
Nikhil Sharma
