Thanks Robbie, Actually we are setting requestTimeout to 60000. We created a support ticket with Microsoft engineering team. And we provided them with a heap dump. I will update here once problem identified. Meanwhile, if you have any suggestion about things to check in the dump or hints to send to Microsoft, that would be great.
Again thanks a lot for your reply, Best regards, Laabidi Raissi C1 - Internal use -----Message d'origine----- De : Robbie Gemmell <robbie.gemm...@gmail.com> Envoyé : mardi 18 juin 2024 12:56 À : users@qpid.apache.org Objet : Re: Producers getting blocked indefinitely EXTERNAL EMAIL: BE VIGILANT The jms.sendTimeout will not cover that particular situation since you are not actually sending in that case, but rather first creating a producer, so also try looking at jms.requestTimeout as well which governs things like session/producer/consumer creation. On Mon, 17 Jun 2024 at 12:21, RAISSI Laabidi - FREELANCE.COM <laabidi.rai...@loreal.com> wrote: > > Hello Robbie, > > Thanks a lot for your reply. > My bad for both the erroneous doc ref and the screenshot. Sorry for that. > I will check the docs to try and set the timeout and if there is any other > config I can set. > As for stacktrace, I am pasting it below. I hope you can spot any suspicious > thing (like for e.g ConservativeProviderFuture being used instead of other > provider ?). > As for producers getting closed from Azure side, like you said, the library > raises a "producer closed" error and we are able to handle that. > > PS: I am sorry if this reply opens a new separate thread. I did subscribe to > the dev list and sent the question to users one. So I didn't get the emails > to reply directly. So I had to fetch the email manually and reply. > > Here is the stacktrace: > > "org.springframework.jms.JmsListenerEndpointContainer#0-5" prio=5 tid=101 > WAITING > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(<unresolved string 0x0>) > at > org.apache.qpid.jms.provider.ConservativeProviderFuture.sync(ConservativeProviderFuture.java:116) > local variable: > org.apache.qpid.jms.provider.ConservativeProviderFuture#1 > local variable: > org.apache.qpid.jms.provider.ConservativeProviderFuture#1 > at > org.apache.qpid.jms.JmsConnection.createResource(JmsConnection.java:693) > local variable: org.apache.qpid.jms.JmsConnection#2 > local variable: > org.apache.qpid.jms.provider.ConservativeProviderFuture#1 > at > org.apache.qpid.jms.JmsMessageProducer.<init>(JmsMessageProducer.java:73) > at org.apache.qpid.jms.JmsSession.createProducer(JmsSession.java:676) > local variable: org.apache.qpid.jms.JmsMessageProducer#1 > at > org.springframework.jms.core.JmsTemplate.doCreateProducer(JmsTemplate.java:1141) > at > org.springframework.jms.core.JmsTemplate.createProducer(JmsTemplate.java:1122) > local variable: org.springframework.jms.core.JmsTemplate#2 > at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:628) > local variable: org.springframework.jms.core.JmsTemplate#2 > local variable: org.apache.qpid.jms.JmsSession#3 > at > org.springframework.jms.core.JmsTemplate.lambda$send$3(JmsTemplate.java:612) > at > org.springframework.jms.core.JmsTemplate$$Lambda$2504+0x00007fe2911046d8.doInJms(<unresolved > string 0x0>) > at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:530) > local variable: org.springframework.jms.core.JmsTemplate#2 > local variable: jdk.proxy3.$Proxy222#1 > local variable: org.apache.qpid.jms.JmsSession#3 > at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:610) > at > org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.ja > va:717) > > Kindly yours, > Laabidi > > > C1 - Internal use > -----Message d'origine----- > De : Robbie Gemmell <robbie.gemm...@gmail.com> Envoyé : lundi 17 juin > 2024 12:34 À : users@qpid.apache.org Objet : Re: Producers getting > blocked indefinitely > > The documentation you referenced is for an older and completely separate > client (for AMQP 0-x, rather than AMQP 1.0 as used by the newer client and > Service Bus), so the details and options contained are not applicable in this > case. > > The docs for the client you are using are at: > https://urldefense.com/v3/__https://qpid.apache.org/releases/qpid-jms- > 2.5.0/docs/__;!!IY5JXqZAIQ!_GOYCcgRTEjN__oXuY2Gvq4zirKkd6YO4nb0VidWhV2 > qcs40i1ZjXJHg0s3xomZ44tBNL46LCNuIDGcvJvPACcJoNPZ9$ > > The send call timeout, which is indeed not set by default, can be controlled > with the jms.sendTimeout URI option: > https://urldefense.com/v3/__https://qpid.apache.org/releases/qpid-jms- > 2.5.0/docs/index.html*jms-configuration-options__;Iw!!IY5JXqZAIQ!_GOYC > cgRTEjN__oXuY2Gvq4zirKkd6YO4nb0VidWhV2qcs40i1ZjXJHg0s3xomZ44tBNL46LCNu > IDGcvJvPACQ_fNl7M$ > > Attachments dont make it to the list. > > ServiceBus has its own particular handling of idle that will e.g close a > producer if not actually sending messages for a certain short period, which > could be interacting with your caching, though in general I'd expect the > client to handle a producer being closed like that and kill the send call, > something we do test and I'm not aware of specific issue with (and hasnt > changed in years). > > > On Mon, 17 Jun 2024 at 10:58, RAISSI Laabidi - FREELANCE.COM > <laabidi.rai...@loreal.com> wrote: > > > > Hi, > > > > We are using Azure Service Bus with QPID JMS version: 2.5.0. > > > > Frequently we are having threads that get stuck in WAITING state when > > sending messages, for indefinite time. Frequently, but not > > deterministically. > > > > I attached a picture of relevant stack trace. > > > > As I understood from this page: > > https://urldefense.com/v3/__https://qpid.apache.org/releases/qpid-jm > > s- > > amqp-0-x-6.4.0/jms-amqp-0-8-book/JMS-Client-0-8-Appendix-ProducerFlo > > wC > > ontrol-Impact.html__;!!IY5JXqZAIQ!_GOYCcgRTEjN__oXuY2Gvq4zirKkd6YO4n > > b0 VidWhV2qcs40i1ZjXJHg0s3xomZ44tBNL46LCNuIDGcvJvPACSKdhKdi$ > > > > Producer may block if queue is overfull (or other issues), and it will > > timeout after “qpid.flow_control_wait_failure” which we are keeping at > > default value (60000ms). But no timeout is happening. > > > > > > > > Curiously, when investigating heap dump, I see that the value of attribute > > “sendTimeout” of connectionInfo (class: JmsConnectionInfo) inside the > > JmsConnection is set to -1 (INFINITE). > > > > Can this be the reason for not timeouting ? And if possible to indicate the > > configuration that is responsible for it. > > > > > > > > Finally, we are calling the client via Spring JmsTemplate. And we are > > using Spring’s CachingConnectionFactory to cache producers. > > > > Unfortunately we can not enable all traces because we are having millions > > of messages and the issue is quite “random” (at least we can not a pattern). > > > > > > > > Please let me know if I can add further details. > > > > > > > > Thanks in advance. > > > > > > > > Kindly yours, > > > > Laabidi > > > > > > > > > > This message and any attachments are confidential and intended solely for > > the addressees. > > If you receive this message in error, please delete it and immediately > > notify the sender. If the reader of this message is not the intended > > recipient, you are hereby notified that any unauthorized use, copying or > > dissemination is prohibited. E-mails are susceptible to alteration. Neither > > LOREAL nor any of its subsidiaries or affiliates shall be liable for the > > message if altered, changed or falsified. > > > > > > C1 - Internal use > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For > > additional commands, e-mail: users-h...@qpid.apache.org > > > This message and any attachments are confidential and intended solely for the > addressees. > If you receive this message in error, please delete it and immediately notify > the sender. If the reader of this message is not the intended recipient, you > are hereby notified that any unauthorized use, copying or dissemination is > prohibited. E-mails are susceptible to alteration. Neither LOREAL nor any of > its subsidiaries or affiliates shall be liable for the message if altered, > changed or falsified. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For > additional commands, e-mail: users-h...@qpid.apache.org > On Mon, 17 Jun 2024 at 12:21, RAISSI Laabidi - FREELANCE.COM <laabidi.rai...@loreal.com> wrote: > > Hello Robbie, > > Thanks a lot for your reply. > My bad for both the erroneous doc ref and the screenshot. Sorry for that. > I will check the docs to try and set the timeout and if there is any other > config I can set. > As for stacktrace, I am pasting it below. I hope you can spot any suspicious > thing (like for e.g ConservativeProviderFuture being used instead of other > provider ?). > As for producers getting closed from Azure side, like you said, the library > raises a "producer closed" error and we are able to handle that. > > PS: I am sorry if this reply opens a new separate thread. I did subscribe to > the dev list and sent the question to users one. So I didn't get the emails > to reply directly. So I had to fetch the email manually and reply. > > Here is the stacktrace: > > "org.springframework.jms.JmsListenerEndpointContainer#0-5" prio=5 tid=101 > WAITING > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(<unresolved string 0x0>) > at > org.apache.qpid.jms.provider.ConservativeProviderFuture.sync(ConservativeProviderFuture.java:116) > local variable: > org.apache.qpid.jms.provider.ConservativeProviderFuture#1 > local variable: > org.apache.qpid.jms.provider.ConservativeProviderFuture#1 > at > org.apache.qpid.jms.JmsConnection.createResource(JmsConnection.java:693) > local variable: org.apache.qpid.jms.JmsConnection#2 > local variable: > org.apache.qpid.jms.provider.ConservativeProviderFuture#1 > at > org.apache.qpid.jms.JmsMessageProducer.<init>(JmsMessageProducer.java:73) > at org.apache.qpid.jms.JmsSession.createProducer(JmsSession.java:676) > local variable: org.apache.qpid.jms.JmsMessageProducer#1 > at > org.springframework.jms.core.JmsTemplate.doCreateProducer(JmsTemplate.java:1141) > at > org.springframework.jms.core.JmsTemplate.createProducer(JmsTemplate.java:1122) > local variable: org.springframework.jms.core.JmsTemplate#2 > at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:628) > local variable: org.springframework.jms.core.JmsTemplate#2 > local variable: org.apache.qpid.jms.JmsSession#3 > at > org.springframework.jms.core.JmsTemplate.lambda$send$3(JmsTemplate.java:612) > at > org.springframework.jms.core.JmsTemplate$$Lambda$2504+0x00007fe2911046d8.doInJms(<unresolved > string 0x0>) > at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:530) > local variable: org.springframework.jms.core.JmsTemplate#2 > local variable: jdk.proxy3.$Proxy222#1 > local variable: org.apache.qpid.jms.JmsSession#3 > at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:610) > at > org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.ja > va:717) > > Kindly yours, > Laabidi > > > C1 - Internal use > -----Message d'origine----- > De : Robbie Gemmell <robbie.gemm...@gmail.com> Envoyé : lundi 17 juin > 2024 12:34 À : users@qpid.apache.org Objet : Re: Producers getting > blocked indefinitely > > The documentation you referenced is for an older and completely separate > client (for AMQP 0-x, rather than AMQP 1.0 as used by the newer client and > Service Bus), so the details and options contained are not applicable in this > case. > > The docs for the client you are using are at: > https://urldefense.com/v3/__https://qpid.apache.org/releases/qpid-jms- > 2.5.0/docs/__;!!IY5JXqZAIQ!_GOYCcgRTEjN__oXuY2Gvq4zirKkd6YO4nb0VidWhV2 > qcs40i1ZjXJHg0s3xomZ44tBNL46LCNuIDGcvJvPACcJoNPZ9$ > > The send call timeout, which is indeed not set by default, can be controlled > with the jms.sendTimeout URI option: > https://urldefense.com/v3/__https://qpid.apache.org/releases/qpid-jms- > 2.5.0/docs/index.html*jms-configuration-options__;Iw!!IY5JXqZAIQ!_GOYC > cgRTEjN__oXuY2Gvq4zirKkd6YO4nb0VidWhV2qcs40i1ZjXJHg0s3xomZ44tBNL46LCNu > IDGcvJvPACQ_fNl7M$ > > Attachments dont make it to the list. > > ServiceBus has its own particular handling of idle that will e.g close a > producer if not actually sending messages for a certain short period, which > could be interacting with your caching, though in general I'd expect the > client to handle a producer being closed like that and kill the send call, > something we do test and I'm not aware of specific issue with (and hasnt > changed in years). > > > On Mon, 17 Jun 2024 at 10:58, RAISSI Laabidi - FREELANCE.COM > <laabidi.rai...@loreal.com> wrote: > > > > Hi, > > > > We are using Azure Service Bus with QPID JMS version: 2.5.0. > > > > Frequently we are having threads that get stuck in WAITING state when > > sending messages, for indefinite time. Frequently, but not > > deterministically. > > > > I attached a picture of relevant stack trace. > > > > As I understood from this page: > > https://urldefense.com/v3/__https://qpid.apache.org/releases/qpid-jm > > s- > > amqp-0-x-6.4.0/jms-amqp-0-8-book/JMS-Client-0-8-Appendix-ProducerFlo > > wC > > ontrol-Impact.html__;!!IY5JXqZAIQ!_GOYCcgRTEjN__oXuY2Gvq4zirKkd6YO4n > > b0 VidWhV2qcs40i1ZjXJHg0s3xomZ44tBNL46LCNuIDGcvJvPACSKdhKdi$ > > > > Producer may block if queue is overfull (or other issues), and it will > > timeout after “qpid.flow_control_wait_failure” which we are keeping at > > default value (60000ms). But no timeout is happening. > > > > > > > > Curiously, when investigating heap dump, I see that the value of attribute > > “sendTimeout” of connectionInfo (class: JmsConnectionInfo) inside the > > JmsConnection is set to -1 (INFINITE). > > > > Can this be the reason for not timeouting ? And if possible to indicate the > > configuration that is responsible for it. > > > > > > > > Finally, we are calling the client via Spring JmsTemplate. And we are > > using Spring’s CachingConnectionFactory to cache producers. > > > > Unfortunately we can not enable all traces because we are having millions > > of messages and the issue is quite “random” (at least we can not a pattern). > > > > > > > > Please let me know if I can add further details. > > > > > > > > Thanks in advance. > > > > > > > > Kindly yours, > > > > Laabidi > > > > > > > > > > This message and any attachments are confidential and intended solely for > > the addressees. > > If you receive this message in error, please delete it and immediately > > notify the sender. If the reader of this message is not the intended > > recipient, you are hereby notified that any unauthorized use, copying or > > dissemination is prohibited. E-mails are susceptible to alteration. Neither > > LOREAL nor any of its subsidiaries or affiliates shall be liable for the > > message if altered, changed or falsified. > > > > > > C1 - Internal use > > > > > > -------------------------------------------------------------------- > > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For > > additional commands, e-mail: users-h...@qpid.apache.org > > > This message and any attachments are confidential and intended solely for the > addressees. > If you receive this message in error, please delete it and immediately notify > the sender. If the reader of this message is not the intended recipient, you > are hereby notified that any unauthorized use, copying or dissemination is > prohibited. E-mails are susceptible to alteration. Neither LOREAL nor any of > its subsidiaries or affiliates shall be liable for the message if altered, > changed or falsified. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For > additional commands, e-mail: users-h...@qpid.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org This message and any attachments are confidential and intended solely for the addressees. If you receive this message in error, please delete it and immediately notify the sender. If the reader of this message is not the intended recipient, you are hereby notified that any unauthorized use, copying or dissemination is prohibited. E-mails are susceptible to alteration. Neither LOREAL nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
