I found out the problem is conditioned to having the row cache enabled. Whenever a query would return an empty result set in a particular table, it would fail instead with the exception being thrown in all all nodes.
Disabling the row cache for that particular table fixes the problem. Re-enabling has not caused the issue again yet. I do not have row cache saving enabled, and the issue persisted between node restarts, which it's somewhat strange. I couldn't reproduce the issue with any other tables. I can produce a sanitized dump of the SSTable if a developer desires. -- Regards, Daniel On Wed, Mar 22, 2017 at 2:33 PM Daniel Miranda <dan...@cobli.co> wrote: > Thank you for the pointer Michael, I'll try to investigate if this is the > same bug I am seeing. > > I am afraid it might not be, since I'm observing the error periodically, > not just during compactions, and the traceback seems different. > > Regards, > Daniel > > On Wed, Mar 22, 2017 at 1:27 PM Michael Shuler <mich...@pbandjelly.org> > wrote: > > Possibly https://issues.apache.org/jira/browse/CASSANDRA-12336, which > shows fixed in 3.0.9, 3.8. There are a couple related bug reports listed > on there, which you might investigate, as well. > > -- > Kind regards, > Michael > > On 03/22/2017 11:21 AM, Daniel Miranda wrote: > > Greetings, > > > > Recently I've started to see the an assertion (traceback follows at the > > end of the message) causing exceptions in a 3-node Cassandra 3.5 cluster > > (running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all > > nodes. Repairs run fine without indicating any errors. > > > > I can't seem to find any information about it from someone else or any > > bug reports. > > > > Should I bother running an SSTable scrub? Is it a known issue that is > > fixed in subsequent versions? > > > > Thanks in advance, > > Daniel > > > > --- > > WARN [SharedPool-Worker-1] 2017-03-16 18:54:35,587 > > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on > > thread Thread[SharedPool-Worker-1,5,main]: {} > > java.lang.AssertionError: null > > at > > > org.apache.cassandra.db.rows.UnfilteredRowIterators.concat(UnfilteredRowIterators.java:157) > > ~[apache-cassandra-3.5.jar:3.5] > > at > > > org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(SinglePartitionReadCommand.java:420) > > ~[apache-cassandra-3.5.jar:3.5] > > at > > > org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:324) > > ~[apache-cassandra-3.5.jar:3.5] > > at > > org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:366) > > ~[apache-cassandra-3.5.jar:3.5] > > at > > > org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1797) > > ~[apache-cassandra-3.5.jar:3.5] > > at > > > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2466) > > ~[apache-cassandra-3.5.jar:3.5] > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > ~[na:1.8.0_111] > > at > > > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) > > ~[apache-cassandra-3.5.jar:3.5] > > at > > > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136) > > [apache-cassandra-3.5.jar:3.5] > > at > > org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) > > [apache-cassandra-3.5.jar:3.5] > > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] > > -- > > *Daniel Miranda* > > > > DevOps Engineering Intern > > (11) 991959845 > > www.cobli.co <http://www.cobli.co/> > >
