For development, I have a single-node Riak 1.4 cluster that I'm
connecting to with the Java client 2.0.1. Both client and server are
running on the same node, connecting via localhost.
Every now and then, an update operation ... hangs. On the console I
see this strange error:
12:21:46.646 [nioEventLoopGroup-2-1] ERROR
com.basho.riak.client.core.RiakNode - Operation onException() channel:
id:-1348399652 localhost:8087 {}
io.netty.util.concurrent.BlockingOperationException:
DefaultChannelPromise@63f65d07(incomplete)
at
io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:396)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:257)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:667)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:636)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:570)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:90)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:201)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:195)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:131)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.core.FutureOperation.setResponse(FutureOperation.java:170)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:824)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:58)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:340)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:326)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:155)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:108)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:340)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:326)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:116)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
12:21:46.653 [nioEventLoopGroup-2-1] WARN
i.n.channel.DefaultChannelPipeline - An exception was thrown by a user
handler's exceptionCaught() method while handling the following
exception:
io.netty.util.concurrent.BlockingOperationException:
DefaultChannelPromise@63f65d07(incomplete)
at
io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:396)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:257)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakNode.doGetConnection(RiakNode.java:667)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakNode.getConnection(RiakNode.java:636)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakNode.execute(RiakNode.java:570)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.core.DefaultNodeManager.executeOnNode(DefaultNodeManager.java:90)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:201)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakCluster.execute(RiakCluster.java:195)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.api.commands.kv.StoreValue.executeAsync(StoreValue.java:117)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.api.commands.kv.UpdateValue$1.handle(UpdateValue.java:182)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.api.commands.ListenableFuture.notifyListeners(ListenableFuture.java:78)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.api.commands.CoreFutureAdapter.handle(CoreFutureAdapter.java:120)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.core.FutureOperation.fireListeners(FutureOperation.java:131)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.core.FutureOperation.setResponse(FutureOperation.java:170)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at com.basho.riak.client.core.RiakNode.onSuccess(RiakNode.java:824)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
com.basho.riak.client.core.netty.RiakResponseHandler.channelRead(RiakResponseHandler.java:58)
~[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:340)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:326)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:155)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.handler.codec.ByteToMessageCodec.channelRead(ByteToMessageCodec.java:108)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:340)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:326)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:116)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101)
[storage-backend-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
VisualVM reports this for the hanging thread:
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <4c34894c> (a java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at
com.basho.riak.client.api.commands.kv.UpdateValue$UpdateValueFuture.await(UpdateValue.java:526)
at com.basho.riak.client.api.RiakCommand.execute(RiakCommand.java:88)
at com.basho.riak.client.api.RiakClient.execute(RiakClient.java:293)
at
com.skalio.skaliopush.services.ManifestFacade.updateMalware(ManifestFacade.java:222)
at com.skalio.skaliopush.tasks.MalwareScanTask.call(MalwareScanTask.java:69)
at com.skalio.skaliopush.tasks.MalwareScanTask.call(MalwareScanTask.java:35)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- locked <617c74e5> (a java.util.concurrent.ThreadPoolExecutor$Worker)
The call to RiakClient.execute() does not catch any Exceptions. There
is absolutely no load on the box when this happens, and it's not
reproducible.
What is happening here? How can I fix it? How can I at least become
aware of the Exception?
Thanks for your help!
Cheers,
Henning
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
