Greetings,
We have an issue where business is reporting the messages are not processing. Upon checking, we saw that messages were sitting in the delivery count which were not browsable in the hawtio jolokia console. At the same time this issue has happened multiple times in both nonprod env as well as prod env. We also know that this issue is fixed when we restart the qpid which is not feasible at all the time given the business impact. After doing analysis on qpid-dispatch logs and activemq broker logs, I found below error/exceptions which occurred starting from June 14th. Prior to June 14th these exceptions did not occur. We are not sure if this is the culprit but require your assistance/guidance on the same. Broker: 1. 2022-09-13 13:16:52,574 WARN [org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler] AMQ119008: error cancelling message AMQPLargeMessage 2. 2022-09-13 15:29:28,585 WARN [org.eclipse.jetty.http.HttpParser] Header is too large 8193>8192 Qpid: 1. 2022-09-13 16:03:47.294355 -0400 SERVER (info) [C4840901] Connection from 10.XX.XX.XX:49995 (to qpid-xx-xx:5001) failed: amqp:connection:framing-error Unknown protocol detected: 'GET / HTTP/1.1\x0d\x0aHost: 10.xx.xx.xx:5001\x0d\x0a\x0d\x0a' 1. 2022-09-14 15:32:46.058562 -0400 ROUTER_CORE (info) [C1356][L42699] Stuck delivery: At least one delivery on this link has been undelivered/unsettled for more than 10 seconds We have below architecture in respect to activemq master/slave setup. MS(kube-microservice) --> NLB(loadblancer) --> Apache QPID (1.16) --> Artemis 2.18 (master/slave) Please also add below two users to Activemq Slack Channel. Thank You 1. ekta.awas...@officedepot.com 2. ajit.tathaw...@officedepot.com Ekta Awasthi, Engineer, EAI Operations & Support | Office Depot, Inc. 6600 North Military Trail | Boca Raton, FL 33496-2434 Office: 561-438-3552 | Mobile: 206-966-5577 | ekta.awas...@officedepot.com<mailto:ekta.awas...@officedepot.com> -- Tips for EAI Support Engagement -- -EAI Pre-Prod Support: Create requests on the following JIRA board EAI Operations Support<https://officedepot.atlassian.net/secure/RapidBoard.jspa?rapidView=823&projectKey=EOS> -EAI Production Support: Create requests via IT Service Desk<https://portal.compucom.com/SSO/Default.aspx?init=2468> self-service portal, instructions click here<https://officedepot.sharepoint.com/sites/portal/TechBytes/EUS/EUS%20User%20Guides/SERVICE%20DESK/RESOLVER%20PORTAL/Self-Service%20Portal%20-%20Resolver%20(IT%20to%20IT)%20Incident%20Request%20Form.pdf>: EAI Support queue --> ODP - Enterprise Apps Integration Support -As a reminder, the Service Availability Managers should be engaged for any service impacting issues, with a ***Page*** to naitavailabilitym...@officedepot.com<mailto:naitavailabilitym...@officedepot.com> or by initiating a MIRT CONFIDENTIALITY NOTICE: The information contained in this email and attached document(s) may contain confidential information that is intended only for the addressee(s). If you are not the intended recipient, you are hereby advised that any disclosure, copying, distribution or the taking of any action in reliance upon the information is prohibited. If you have received this email in error, please immediately notify the sender and delete it from your system.
