one query in our prod system under heavy load failed. I cant tell which query it was. But I don't think we were using an invalid consistency level since our configuration work all the time except when it goes under heavy load. If I can reproduce this I will pass more information.
I wonder if this is due to a race condition in Paxos implementation. I found the following in the cassandra's log file: ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794 >> QueryMessage.java (line 131) Unexpected error during query > > java.lang.UnsupportedOperationException: Invalid consistency level: >> LOCAL_SERIAL > > at >> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137) > > at >> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416) > > at >> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228) > > at >> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423) > > at >> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380) > > at >> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > > at >> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) > > at >> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119) > > at >> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > > at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > > at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > > at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > > at >> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > > at >> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > > at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > > ERROR [Native-Transport-Requests:457009] 2014-05-02 13:49:09,794 >> ErrorMessage.java (line 222) Unexpected exception during request > > java.lang.UnsupportedOperationException: Invalid consistency level: >> LOCAL_SERIAL > > at >> org.apache.cassandra.db.ConsistencyLevel.blockFor(ConsistencyLevel.java:137) > > at >> org.apache.cassandra.service.StorageProxy.beginAndRepairPaxos(StorageProxy.java:416) > > at >> org.apache.cassandra.service.StorageProxy.cas(StorageProxy.java:228) > > at >> org.apache.cassandra.cql3.statements.ModificationStatement.executeWithCondition(ModificationStatement.java:423) > > at >> org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:380) > > at >> org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188) > > at >> org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:222) > > at >> org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119) > > at >> org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:304) > > at >> org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > > at >> org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) > > at >> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) > > at >> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43) > > at >> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67) > > at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:744) > >
