What you're seeing is the expected behavior in certain circumstances [1]. In your case, you're sending a durable message to the broker and the connection fails. The problem now is that the client doesn't know what happened to the message. Did it make it to the broker and get persisted to disk before the connection failed (i.e. before the ack is sent back to the producer)? If so, then retrying the send operation will result in a duplicate message. Or did the connection fail before the message made it to the broker? If so, then *not* retrying the send operation will mean the message is lost. Since there's no clear way to proceed it simply throws an exception, and it's up to the application to decide what to do based on its use-case. Maybe the consumers are idempotent so duplicates don't matter. Maybe the producer is using duplicate detection [2] so it can resend the message without any worries. Maybe a few lost messages don't matter so it can just ignore the exception. In short, you need to handle such failures as necessary.
Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/ha.html#handling-blocking-calls-during-failover [2] https://activemq.apache.org/components/artemis/documentation/latest/duplicate-detection.html On Thu, Mar 23, 2023 at 3:27 PM John Lilley <john.lil...@redpointglobal.com.invalid> wrote: > We’ve added a chaos monkey to randomly disconnect Artemis clients from > broker. > > This mostly works, however, we do get occasional errors that are not > retried by the AMQ JMS client. > > This manifests with the exception: AMQ219016: Connection failure detected. > Unblocking a blocking call that will never get a response. > > I don’t know why the AMQ client thinks it will never get a response. > Shouldn’t the client reconnect and continue without error? Do we need to > retry explicitly? > > > > This is using version 2.28.0. Example stack trace: > > > > 2023-03-23T14:19:54.666 [Schedule-Runner] LazyStartPeriodicThread.run:97 > [] ERROR - Exception in interval action RpcException: httpCode=500, > errorMessage=error sending message: AMQ219016: Connection failure detected. > Unblocking a blocking call that will never get a response > > errorFrames: > > javax.jms.JMSException: AMQ219016: Connection failure detected. Unblocking > a blocking call that will never get a response > > at > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558) > > at > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450) > > at > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129) > > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483) > > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221) > > at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351) > > at > net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98) > > at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165) > > at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231) > > at > net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178) > > at > net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205) > > at > net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387) > > at > net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325) > > at > net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511) > > at net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510) > > at > net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95) > > at java.base/java.lang.Thread.run(Thread.java:833) > > Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED > message=AMQ219016: Connection failure detected. Unblocking a blocking call > that will never get a response] > > ... 28 more > > Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED > message=AMQ219006: Channel disconnected] > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383) > > at > org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > at > net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:295) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98) > > at > net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165) > > at > net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231) > > at > net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178) > > at > net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205) > > at > net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387) > > at > net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325) > > at > net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511) > > at > net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510) > > at > net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95) > > at java.base/java.lang.Thread.run(Thread.java:833) > > Caused by: javax.jms.JMSException: AMQ219016: Connection failure detected. > Unblocking a blocking call that will never get a response > > at > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558) > > at > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450) > > at > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129) > > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483) > > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221) > > at > net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351) > > at > net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289) > > ... 17 more > > Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED > message=AMQ219016: Connection failure detected. Unblocking a blocking call > that will never get a response] > > ... 28 more > > Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED > message=AMQ219006: Channel disconnected] > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383) > > at > org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > 2023-03-23T14:19:54.700 [Thread-20] JmsStaticConnectionPool.onException:78 > [] ERROR - Receive error occurred. javax.jms.JMSException: > ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: > Channel disconnected] > > at > org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714) > > at > org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:764) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:752) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:707) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED > message=AMQ219006: Channel disconnected] > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383) > > at > org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240) > > ... 6 more > > > > [image: rg] <https://www.redpointglobal.com/> > > John Lilley > > Data Management Chief Architect, Redpoint Global Inc. > > 888 Worcester Street, Suite 200 Wellesley, MA 02482 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. >
