Hello, We are using ActiveMQ Artemis to enable multiple web applications of the same type (developed with Spring Boot) to send messages with each other to keep data up to date. When working in a development environment such as Eclipse/IntelliJ IDEA, we are able to have these instances communicate with each other without issues. However, we encounter issues when we deploy the application on Wildfly in both development and production machines.
In both environments, there is a server hosting the Artemis broker, and there are other servers, each hosting various instances of the application. The problem we encounter is the loss of some messages, which end up in the DLQ (Dead Letter Queue). This issue particularly occurs as network traffic increases, especially on queues that accumulate hundreds or sometimes thousands of messages. Upon examining the Artemis logs, this is what is reported for each lost message: WARN [org.apache.activemq.artemis.core.server] AMQ222149: Message Reference[8970196914]:RELIABLE:CoreMessage[messageID=8970196914,durable=true,userID=1adf1f6d-119a-11ee-93f5-005056bbcfcf,priority=4, timestamp=Fri Jun 23 09:46:46 CEST 2023,expiration=0, durable=true, address=I01-or,size=35909,properties=TypedProperties[__AMQ_CID=ID:A12.sg.it-34045-1681291484686-0:1,_AMQ_GROUP_SEQUENCE=0,__HDR_BROKER_IN_TIME=1687506406853,_AMQ_ROUTING_TYPE=1,__HDR_ARRIVAL=0,__HDR_COMMAND_ID=192161375,__HDR_PRODUCER_ID=[0000 0040 7B01 002C 4944 3A41 4947 4F52 2D31 322E 736F 6769 6E2E 6974 2D33 ... 3132 3931 3438 3436 3836 2D31 3A31 0000 0000 0000 0001 0000 0000 0298 3F1A),_AMQ_DUPL_ID=ID:A12.sg.it-34045-1681291484686-1:1:43532058:1:1,__HDR_MESSAGE_ID=[0000 0053 6E00 017B 0100 2C49 443A 4149 474F 522D 3132 2E73 6F67 696E 2E69 ... 00 0000 0000 0100 0000 0002 983F 1A00 0000 0000 0000 0100 0000 0000 0000 00),__HDR_DROPPABLE=false]]@1995702679 has reached maximum delivery attempts, sending it to Dead Letter Address DLQ from I01-or As indicated by the logs, the maximum number of attempts to recover the message has been reached. However, upon analyzing the application logs, there are no errors or any information indicating an attempt by the application to retrieve the message (for each received message, logs are generated at the time of message reception). Additionally, there are no messages indicating connection issues to the Artemis broker. We attempted to replicate the same scenario locally by using a Docker container for Artemis and a virtual machine to deploy the application (with the same OS as the development and production machines). When deploying the application on Wildfly we encounter a similar issue as described earlier. However, if we try to run the application from the virtual machine but in a development environment (IntelliJ IDEA), the issue does not occur (we performed the same operations in both cases and used the same configuration to connect to the Artemis broker). Regarding the versions used, we provide them below: * ActiveMQ Artemis, version 2.16.0. We also tried using a more recent version of Artemis (2.30.0), but we have the same problem. * Web Application: Spring Boot, Java 8, Tomcat web server. * Database: PostgreSQL version 10.10. * Wildfly 18.0.0. As for the configuration parameters used for Artemis, the broker.xml configuration file has not been modified. The configuration takes place within the application. We are using the starter ‘spring-boot-starter-activemq’ with the following settings: * Artemis Mode: native. * Connection Factory - trust al packages: true. * Connection Factory – connection response timeout: 999999. * Connection Factory – close timeout: 999999. * Connection Factory – send Timeout: 999999. * Connection Factory – use async send: true. * Redelivery Policy - initial redelivery delay: 1000. * Redelivery Policy - back off multiplier: 3. * Redelivery Policy - max redeliveries: 7. * Redelivery Policy - max redelivery delay: 60000. * Default JMS Container Factory – recovery interval: 5000. What could be the cause of the problem, and how can we resolve it? Thank you, Christian Esposito
