Yes, I too faced similar issues while randomly killing artemis instances. The pattern which I observed was mainly, when you kill live instances in a cluster w/o running backup node. So, in our case we incorporated *client side failover* detection. IMO, most of the times; failover happens transparently when going from live to backup nodes. But at the same time, we couldn't replicate it everytime. So, to keep things save; it is better to have a safety net designed.
Regards, Prateek Jain -------------------------------------------------------------- EXPECTATION : Causes all troubles...... -------------------------------------------------------------- On Thu, Mar 23, 2023 at 8:27 PM John Lilley <john.lil...@redpointglobal.com.invalid> wrote: > We’ve added a chaos monkey to randomly disconnect Artemis clients from > broker. > > This mostly works, however, we do get occasional errors that are not > retried by the AMQ JMS client. > > This manifests with the exception: AMQ219016: Connection failure detected. > Unblocking a blocking call that will never get a response. > > I don’t know why the AMQ client thinks it will never get a response. > Shouldn’t the client reconnect and continue without error? Do we need to > retry explicitly? > > > > This is using version 2.28.0. Example stack trace: > > > > 2023-03-23T14:19:54.666 [Schedule-Runner] LazyStartPeriodicThread.run:97 > [] ERROR - Exception in interval action RpcException: httpCode=500, > errorMessage=error sending message: AMQ219016: Connection failure detected. > Unblocking a blocking call that will never get a response > > errorFrames: > > javax.jms.JMSException: AMQ219016: Connection failure detected. Unblocking > a blocking call that will never get a response > > at > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558) > > at > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450) > > at > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129) > > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483) > > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221) > > at net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351) > > at > net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98) > > at net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165) > > at net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231) > > at > net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178) > > at > net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205) > > at > net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387) > > at > net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325) > > at > net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511) > > at net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510) > > at > net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95) > > at java.base/java.lang.Thread.run(Thread.java:833) > > Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED > message=AMQ219016: Connection failure detected. Unblocking a blocking call > that will never get a response] > > ... 28 more > > Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED > message=AMQ219006: Channel disconnected] > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383) > > at > org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > at > net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:295) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClientChannel.java:217) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientChannel.java:194) > > at > net.redpoint.ipc.jms.JmsRpcClientChannel.call(JmsRpcClientChannel.java:98) > > at > net.redpoint.ipc.clients.RpcClientBase._sync(RpcClientBase.java:165) > > at > net.redpoint.ipc.clients.RpcClientBase._rpc(RpcClientBase.java:231) > > at > net.redpoint.rpdm.ipc.clients.PermissionsClient.doesUserHavePermission(PermissionsClient.java:178) > > at > net.redpoint.rpdm.services.SecurityControlImpl.doesUserHavePermission(SecurityControlImpl.java:205) > > at > net.redpoint.ipc.SecurityControl.checkUserHasPermission(SecurityControl.java:387) > > at > net.redpoint.rpdm.ipc.servers.SchedulerServer.checkUserHasPermission(SchedulerServer.java:325) > > at > net.redpoint.rpdm.ipc.servers.SchedulerServer.list(SchedulerServer.java:264) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl._doSchedule(SchedulerlServerImpl.java:407) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl.doSchedule(SchedulerlServerImpl.java:360) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.lambda$action$0(SchedulerlServerImpl.java:511) > > at > net.redpoint.ipc.SecurityControl.runAsNoThrow(SecurityControl.java:315) > > at > net.redpoint.rpdm.scheduler.SchedulerlServerImpl$Runner.action(SchedulerlServerImpl.java:510) > > at > net.redpoint.rpdm.services.LazyStartPeriodicThread.run(LazyStartPeriodicThread.java:95) > > at java.base/java.lang.Thread.run(Thread.java:833) > > Caused by: javax.jms.JMSException: AMQ219016: Connection failure detected. > Unblocking a blocking call that will never get a response > > at > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:558) > > at > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:450) > > at > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendFullMessage(ActiveMQSessionContext.java:583) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:305) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:277) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:147) > > at > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129) > > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx(ActiveMQMessageProducer.java:483) > > at > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:221) > > at > net.redpoint.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:351) > > at > net.redpoint.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool.java:289) > > ... 17 more > > Caused by: ActiveMQUnBlockedException[errorType=UNBLOCKED > message=AMQ219016: Connection failure detected. Unblocking a blocking call > that will never get a response] > > ... 28 more > > Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED > message=AMQ219006: Channel disconnected] > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383) > > at > org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > > > 2023-03-23T14:19:54.700 [Thread-20] JmsStaticConnectionPool.onException:78 > [] ERROR - Receive error occurred. javax.jms.JMSException: > ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219006: > Channel disconnected] > > at > org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:714) > > at > org.apache.activemq.artemis.jms.client.ActiveMQConnection$JMSFailureListener.connectionFailed(ActiveMQConnection.java:735) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:764) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:752) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:707) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:534) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:527) > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$1.run(ClientSessionFactoryImpl.java:390) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) > > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) > > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) > > Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED > message=AMQ219006: Channel disconnected] > > at > org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:383) > > at > org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:1240) > > ... 6 more > > > > [image: rg] <https://www.redpointglobal.com/> > > John Lilley > > Data Management Chief Architect, Redpoint Global Inc. > > 888 Worcester Street, Suite 200 Wellesley, MA 02482 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. >
