We’ve added a chaos monkey to randomly disconnect Artemis clients from broker.
This mostly works, however, we do get occasional errors that are not retried by
the AMQ JMS client.
This manifests with the exception: AMQ219016: Connection failure detected.
Unblocking a blocking call that will never get a response.
I don’t know why the AMQ client thinks it will never get a response. Shouldn’t
the client reconnect and continue without error? Do we need to retry
explicitly?
This is using version 2.28.0. Example stack trace:
2023-03-23T14:19:54.666 [Schedule-Runner] LazyStartPeriodicThread.run:97 []
ERROR - Exception in interval action RpcException: httpCode=500,
errorMessage=error sending message: AMQ219016: Connection failure detected.
Unblocking a blocking call that will never get a response
errorFrames:
javax.jms.JMSException: AMQ219016: Connection failure detected. Unblocking a
blocking call that will never get a response
at
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558)
at
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
at
org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
at
org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
at
org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351)
at net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289)
at
net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217)
at
net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194)
at net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98)
at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165)
at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231)
at
net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178)
at
net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205)
at
net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387)
at
net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325)
at net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264)
at
net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407)
at
net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360)
at
net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511)
at net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315)
at
net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510)
at
net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED message=AMQ219016:
Connection failure detected. Unblocking a blocking call that will never get a
response]
... 28 more
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
message=AMQ219006: Channel disconnected]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
at
org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
at
net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:295)
at
net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217)
at
net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194)
at
net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98)
at
net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165)
at
net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231)
at
net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178)
at
net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205)
at
net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387)
at
net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325)
at
net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264)
at
net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407)
at
net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360)
at
net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511)
at
net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315)
at
net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510)
at
net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: javax.jms.JMSException: AMQ219016: Connection failure detected.
Unblocking a blocking call that will never get a response
at
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558)
at
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450)
at
org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147)
at
org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
at
org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483)
at
org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221)
at
net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351)
at
net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289)
... 17 more
Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED message=AMQ219016:
Connection failure detected. Unblocking a blocking call that will never get a
response]
... 28 more
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
message=AMQ219006: Channel disconnected]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
at
org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
2023-03-23T14:19:54.700 [Thread-20] JmsStaticConnectionPool.onException:78 []
ERROR - Receive error occurred. javax.jms.JMSException:
ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006:
Channel disconnected]
at
org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714)
at
org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:764)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:752)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:707)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527)
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED
message=AMQ219006: Channel disconnected]
at
org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383)
at
org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240)
... 6 more
[rg] <https://www.redpointglobal.com/>
John Lilley
Data Management Chief Architect, Redpoint Global Inc.
888 Worcester Street, Suite 200 Wellesley, MA 02482
M: +1 7209385761<tel:+1%207209385761> |
john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com>
PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential
and is intended solely for the use of the individual(s) to whom it is
addressed. If you believe you received this e-mail in error, please notify the
sender immediately, delete the e-mail from your computer and do not copy, print
or disclose it to anyone else. If you properly received this e-mail as a
customer, partner or vendor of Redpoint, you should maintain its contents in
confidence subject to the terms and conditions of your agreement(s) with
Redpoint.
