> After ten seconds, the packet appears to be sent, but we log its delivery in our service.
What logging do you see that indicates the packet was sent? To be clear, if the client doesn't receive an acknowledgement from the broker (e.g. due to a network issue, broker crash, etc.) that the message was sent (i.e. a response to a packet 71) then the client can only conclude that the message was not sent. > But then a full minute passes before the original (sending) client resumes communication with the broker. Do you have any idea what's happening during this 60-second window? Have you, for example, gathered any thread dumps from the client during this time to see exactly what it's doing? If so, what did they show? If not, could you? > We want to shorten this one-minute reconnect interval, ideally to < 30 seconds. What combination of settings will do this? I don't think I can say exactly what settings need to change (if any) without understanding where the delay is happening. I'd just be guessing at this point. It could be something at the messaging layer (e.g. in the Artemis client) or it could be something in the transport layer (e.g. operating system TCP setting). I've seen weird stuff sometimes with Kubernetes since it tends to deconstruct stuff that's typically assumed good in normal deployments (e.g. network, OS, etc.). That said, it would be worth knowing *all* the connection-related parameters you're setting on the client. Can you provide that info? Justin On Wed, Nov 20, 2024 at 8:36 AM John Lilley <john.lil...@redpointglobal.com> wrote: > Hi Justin, > > I appreciate you taking the time. > > We've been hitting this issue in various forms for a while. There appear > to be two conditions in which we see this: > - The active member of our HA broker pair is moved by Kubernetes and fails > over to the backup > - The connection between our AMQ client and the broker is interrupted. > > When either of these happen, we first see a 10-second delay before the > client emits this error: > AMQ219014: Timed out after waiting 10000 ms for response when sending > packet > After ten seconds, the packet appears to be sent, but we log its delivery > in our service. > But then a full minute passes before the original (sending) client resumes > communication with the broker. > > We want to shorten this one-minute reconnect interval, ideally to < 30 > seconds. What combination of settings will do this? > > Thanks > john > > -----Original Message----- > From: Justin Bertram <jbert...@apache.org> > Sent: Tuesday, November 19, 2024 2:52 PM > To: users@activemq.apache.org > Subject: Re: Warning around reconnect attempt > > *** [Caution] This email is from an external source. Please use caution > responding, opening attachments or clicking embedded links. *** > > > > 1- Should we be setting connectionFactory.setConfirmationWindowSize()? > > The docs for confirmationWindowSize are here [1]. Whether you should set > this parameter is really up to you and the behavior you want for your > application. > > > 2- Are the retry settings reasonable? > > "Reasonable" is hard to judge in a support context like this. I'm not sure > we can really judge. > > > Ultimately we just want to reconnect in under a minute and avoid our > timeout. > > It might be worth investigating why you got the timeout in the first place. > Do you have any data from the broker or the network that might suggest > what happened? > > > Or, maybe we just need to set our timeout longer? > > Without knowing the root cause of the timeout it's impossible to say. > > > Justin > > [1] > > https://urldefense.proofpoint.com/v2/url?u=https-3A__activemq.apache.org_components_artemis_documentation_latest_client-2Dfailover.html-23reconnect-2Dto-2Dthe-2Dsame-2Dserver&d=DwIFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=VpU5fPwEQ8wKbKY5SBkV1DYypS_WKWolD9Eue6u6sYo&m=Mist2dGO6lZfrTPPt2xcN8QXDIwlWhN4ks51T71JNe_2PLfyqLmfUjjOIR6C1_Gb&s=YmllMfJSJOseAoxT8ojTHhtLr2Q3cb3zuZD7KZ2rgAM&e= > > On Tue, Nov 19, 2024 at 3:30 PM John Lilley < > https://urldefense.proofpoint.com/v2/url?u=http-3A__john.lilley-40redpointglobal.com&d=DwIFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=VpU5fPwEQ8wKbKY5SBkV1DYypS_WKWolD9Eue6u6sYo&m=Mist2dGO6lZfrTPPt2xcN8QXDIwlWhN4ks51T71JNe_2PLfyqLmfUjjOIR6C1_Gb&s=CovnqP20ifJlvniztfu3b6uty4AI-pRicMky6zqEi80&e= > > > wrote: > > > 2024-11-19 14:19:51,426 WARN > > [org.apache.activemq.artemis.core.server] > > AMQ222066: Reattach request from 10.147.133.72:53082 failed as there > > is no confirmationWindowSize configured, which may be ok for your > > system > > > > This was issued by the broker at the same time that our JMS client > > issued this warning: > > > > 2024-11-19T14:19:51.417 [Thread-2 (ActiveMQ-client-global-threads)] > > .client.connectionFailureDetected:293 [] WARN - AMQ212037: Connection > > failure to dm-activemq-live-svc/10.0.219.91:61616 has been detected: > > AMQ219014: Timed out after waiting 10000 ms for response when sending > > packet 71 [code=CONNECTION_TIMEDOUT] > > 2024-11-19T14:19:51.414 [main] .JmsProducerPool.send_:389 > > [y6y5v5wm1s0x] WARN - Error sending message, will retry > jakarta.jms.JMSException: > > AMQ219014: Timed out after waiting 10000 ms for response when sending > > packet 71 at > > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBl > > ocking(ChannelImpl.java:550) > > at > > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBl > > ocking(ChannelImpl.java:452) > > at > > org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionCon > > text.sendFullMessage(ActiveMQSessionContext.java:588) > > at > > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.sendRe > > gularMessage(ClientProducerImpl.java:305) > > at > > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend > > (ClientProducerImpl.java:277) > > at > > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(C > > lientProducerImpl.java:147) > > at > > org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(C > > lientProducerImpl.java:129) > > at > > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.doSendx > > (ActiveMQMessageProducer.java:483) > > at > > org.apache.activemq.artemis.jms.client.ActiveMQMessageProducer.send(Ac > > tiveMQMessageProducer.java:221) > > at > > net.redpoint.rpdm.ipc.jms.JmsProducerPool.send_(JmsProducerPool.java:3 > > 85) > > at > > net.redpoint.rpdm.ipc.jms.JmsProducerPool.sendRequest(JmsProducerPool. > > java:300) > > at > > net.redpoint.rpdm.ipc.jms.JmsRpcClientChannel.sendRequest(JmsRpcClient > > Channel.java:230) > > at > > net.redpoint.rpdm.ipc.jms.JmsRpcClientChannel.invokeRaw(JmsRpcClientCh > > annel.java:203) > > Caused by: > > ActiveMQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT > > message=AMQ219014: Timed out after waiting 10000 ms for response when > > sending packet 71] ... 13 more > > > > After 72 seconds, we hit our own 60-second timeout and fail. > > > > In our connection factory, we are setting > > > > connectionFactory.setRetryInterval(1000); > > connectionFactory.setRetryIntervalMultiplier(1.5); > > connectionFactory.setMaxRetryInterval(4000); > > > > > > So my questions are: > > 1- Should we be setting connectionFactory.setConfirmationWindowSize()? > > 2- Are the retry settings reasonable? > > > > Ultimately we just want to reconnect in under a minute and avoid our > > timeout. Or, maybe we just need to set our timeout longer? > > > > Our maven dependency for the JMS driver: > > <dependency> > > <groupId>org.apache.activemq</groupId> > > <artifactId>artemis-jakarta-client</artifactId> > > <version>2.32.0</version> > > </dependency> > > > > Please advise. > > Thanks > > John > > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. ("Redpoint") is > > confidential and is intended solely for the use of the individual(s) > > to whom it is addressed. If you believe you received this e-mail in > > error, please notify the sender immediately, delete the e-mail from > > your computer and do not copy, print or disclose it to anyone else. If > > you properly received this e-mail as a customer, partner or vendor of > > Redpoint, you should maintain its contents in confidence subject to > > the terms and conditions of your agreement(s) with Redpoint. > > > > > ---------- > > This email has been scanned for spam and viruses by Proofpoint Essentials. > Visit the following link to report this email as spam: > > https://us5.proofpointessentials.com/app/report_spam.php?mod_id=11&mod_option=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5fc93a62ffd2f5b082f83c0a4b514b745939f337fbf3a27d426c9d84d149c1c8af15847b3f1403112a84fa6f63ecbe87688a9fdca2e37c7058e22d7eff721005809f48808015c98a65b7f60c124567c12f1f08b8c2e2daace4fc243cf762f0379e29d82d63113e53443993ab7ead863ca1bbe02d58a01b07e01548c0b87bc0888ecd7937a1df8e2d9efb2c4538ffe54c899ba63134a975b990 > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org > For additional commands, e-mail: users-h...@activemq.apache.org > For further information, visit: https://activemq.apache.org/contact > >
