I'm still struggling with a BlockingOperationException thrown by
riak-java-client 2.0.6, which occurs when I put heavy load on Riak KV.
Since https://github.com/basho/riak-java-client/issues/523 is fixed,
this happens only in - what I assume is - an overload-scenario.
The exception:
2016-07-13 14:41:12.789 localhost [nioEventLoopGroup-2-2] ERROR
com.basho.riak.client.core.RiakNode - Operation onException() channel:
id:237445453 localhost:8087 {}
io.netty.util.concurrent.BlockingOperationException:
DefaultChannelPromise@77ccd827(incomplete)
at
io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)
at
io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
at
io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)
at
io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
at
io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
at
com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:697)
at com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:656)
at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:587)
at
com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:91)
at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:322)
at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:240)
at
com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117)
at
com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182)
at
com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78)
at
com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120)
at
com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:176)
at
com.basho.riak.client.core.FutureOperation.setComplete(FutureOperation.java:224)
at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:878)
at
com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:30)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:276)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:263)
at
io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:103)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Then shortly thereafter:
2016-07-13 14:41:12.820 localhost [nioEventLoopGroup-2-2] ERROR
com.basho.riak.client.core.RiakNode - Write failed on RiakNode
localhost:8087 id: 237445453; cause: {}
java.nio.channels.ClosedChannelException: null
2016-07-13 14:41:12.843 localhost [nioEventLoopGroup-2-2] ERROR
com.basho.riak.client.core.RiakNode - Write failed on RiakNode
localhost:8087 id: 237445453; cause: {}
java.nio.channels.ClosedChannelException: null
The result is a hanging thread, never returning from the call to
`RiakClient.execute()`. It would be great, if in such a case, the
client threw a RiakException (as javadoc says).
But in the meantime, I'm trying to avoid getting into this scenario in
the first place, by sizing the maxConnections of RiakConnector and my
applications frontend threadpools appropriately. My current
observation is this: If I set RiakConnector.withMaxConnections() too
small for the given load (let's say 100), I get the
BlockingOperationException, although according to my (potentially
unprecise connection monitoring), the threshold is never reached.
Setting maxConnections to a higher value (let's say 160) lets the
loadtest run through fine, with around 110 active connections to Riak.
It seems that in order to avoid the BlockingOperationException, I must
set the maxConnections limit higher than what I'll need. Maybe I
should skip maxConnections altogether then, and leave it at 0? But
then, how do I protect against an overload scenario? How many active,
busy connections does Riak KV support? I'm pretty sure the answer to
that is "it depends...".
Thanks for your input.
Cheers,
Henning
--
My other signature is a regular expression.
http://www.pray4snow.de
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
