Hi, Could anyone please have any idea/suggestion about this issue ?
Mit freundlichen Grüßen Gorav Singh externe Fachkraft von der Firma Coforge Limited im Auftrag der DB Systel GmbH SEZ Developer Unit Plot No. TZ-2 & 2A, Sector Tech Zone, Greater Noida, UP 201308 E-Mail: gorav.singh-ext...@deutschebahn.com -----Original Message----- From: Gorav Singh -Extern Sent: Tuesday, August 1, 2023 4:15 PM To: users@activemq.apache.org Subject: RE: Messages got stuck in ActiveMQ 5.18.2 until restart of server Hi Matt/Everyone, I hope that you are doing great ! We upgraded our ActiveMQ version to 5.18.2 but still this issue is coming. I have attached the dependency tree of consumer app and as you suggested in trailing email point 1, we have activemq-client dependency as well under activemq-broker:jar. Should I try to exclude this jar in pom.xml ? Actually we upgraded this consumer application from JDK11 to JDK17 and this issue was somehow coming with JDK11 as well so might be something else but not activemq-client issue (just a guess). Could anyone please suggest any solution for this issue ? Thanks a lot in Advance ! Mit freundlichen Grüßen Gorav Singh externe Fachkraft von der Firma Coforge Limited im Auftrag der DB Systel GmbH SEZ Developer Unit Plot No. TZ-2 & 2A, Sector Tech Zone, Greater Noida, UP 201308 E-Mail: gorav.singh-ext...@deutschebahn.com -----Original Message----- From: Matt Pavlovich <mattr...@gmail.com> Sent: Friday, June 30, 2023 10:38 PM To: users@activemq.apache.org Subject: Re: Messages got stuck in ActiveMQ 5.18.0 until restart of server [You don't often get email from mattr...@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Hi Gorav- 1. Review dependency tree of the consumer app. Make sure there are only jakarta versions of activemq-client-jakarta and NOT the activemq-client dependency jars. I suspect you have both jars included transitively since you have mentioned including activemq-pool. 2. The activemq-client-jakarta jar in 5.18.0 literally _can't_ work, since some packaging was missing. That is what led to the v5.18.1 release. 3. A thread dump from the Consumer app when the scenario occurs would be the best starting point to solve for why the consumer code is hanging onto messages. 4. Prefetch comes into play, but is I wouldn't change it until you solve for the core issue. Thanks, Matt Pavlovich > On Jun 30, 2023, at 10:34 AM, Gorav Singh -Extern > <gorav.singh-ext...@deutschebahn.com> wrote: > > Hi Matt, > > Thanks for your reply ! Sure, I will use 5.18.1 version but I am not getting > any exception like timeout for hung consumer in the consumer app code. > Is prefetch limit also play role here ? it is default as of now. Should we > try to reduce this limit as well ? > Any suggestion/help is much appreciated. > Thanks a lot in Advance ! > > Mit freundlichen Grüßen > > Gorav Singh > > externe Fachkraft von der Firma Coforge Limited im Auftrag der DB > Systel GmbH SEZ Developer Unit Plot No. TZ-2 & 2A, Sector Tech Zone, > Greater Noida, UP 201308 > E-Mail: gorav.singh-ext...@deutschebahn.com > > -----Original Message----- > From: Matt Pavlovich <mattr...@gmail.com> > Sent: Friday, June 30, 2023 8:10 PM > To: users@activemq.apache.org > Subject: Re: Messages got stuck in ActiveMQ 5.18.0 until restart of > server > > [You don't often get email from mattr...@gmail.com. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > Hi Gorav- > > For sure, you'll need to use the activemq-client-jakarta 5.18.1 jar. The > v5.18.0 activemq-client-jakarta had a minor packaging bug that caused it not > to work at all, so I'm not sure how that client would be working in a happy > path. Also, the PooledConnectionFactory does not have jakarta support yet > (jakarta support is coming in v5.19.x), so there may be a couple of > structural issues going on within the app to start. > > The scenario you are describing aligns most closely to an app coding bug > where the application consumer is hung, and not acknowledging the messages. > The broker dispatched messages to that consumer, so it is not passing them to > another consumer. Restarting the POD causes the Consumer to go away, and that > allows the broker to dispatch those messages to another consumer. > > Look for the queue metric 'InFlightCount'. If that value is about the number > of messages not dequeuing, then most likely the issue is in the consumer app > code. > > Thanks, > Matt Pavlovich > >> On Jun 30, 2023, at 2:02 AM, Gorav Singh -Extern >> <gorav.singh-ext...@deutschebahn.com> wrote: >> >> Hi Team, >> >> We are using activemq-client-jakarta 5.18.0, activemq-broker 5.18.0 with >> spring boot 3.0 and jdk 17. We are consuming messages using @JmsListener >> initially having 1-5 concurrency. >> We are consuming mostly 2500-3000 messages from the queue as the part of >> some campaigns. >> >> >> Now it works fine most of the days but once or twice in a week, we found >> that some of the messages got stuck in the queue which are not consumed but >> messages which pushed later got consumed successfully. >> We have to restart the pod every time in such cases to receive stuck >> messages. >> >> We tried some solution like not to use PooledConnectionFactory for >> consumers, also we changed concurrency setting to 5-5 so that consumer could >> not be killed but nothing worked. >> >> Could you please help me to find out the solution or is it a known >> ActiveMQ issue ? (please note that in JMS configuration, we are using >> ActiveMQConnectionFactory, DefaultJmsListenerContainerFactory, with >> JmsTemplate) >> >> >> Mit freundlichen Grüßen >> >> Gorav Singh >> >> externe Fachkraft von der Firma Coforge Limited im Auftrag der DB >> Systel GmbH SEZ Developer Unit Plot No. TZ-2 & 2A, Sector Tech Zone, >> Greater Noida, UP 201308 >> E-Mail: >> gorav.singh-ext...@deutschebahn.com<mailto:gorav.singh-extern@deutsch >> e >> bahn.com> >> >> >> ________________________________ >> >> Pflichtangaben >> anzeigen<https://www.deutschebahn.com/pflichtangaben/20230607> >> >> Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie >> hier: https://www.deutschebahn.com/de/konzern/datenschutz > > > ________________________________ > > Pflichtangaben > anzeigen<https://www.deutschebahn.com/pflichtangaben/20230607> > > Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie > hier: https://www.deutschebahn.com/de/konzern/datenschutz ________________________________ Pflichtangaben anzeigen<https://www.deutschebahn.com/pflichtangaben/20230717> Nähere Informationen zur Datenverarbeitung im DB-Konzern finden Sie hier: https://www.deutschebahn.com/de/konzern/datenschutz
