Couple of questions: - Version 2.6.13 of the spring-boot-starter-artemis Maven component uses artemis-jms-client 2.19.1. Have you tried upgrading this to a later version? - Have you tried adjusting the minLargeMessageSize URL parameter on your clients so that *no* message is actually considered "large"? This would use more memory on the broker and therefore wouldn't necessarily be recommended, but it would be worth testing to conclusively isolate the problem to "large" messages. - I see that you tried adjusting amqpMinLargeMessageSize, but that only applies to clients using AMQP. Are you using any AMQP clients? I'm guessing you aren't since you didn't see any change in behavior after adjusting that parameter. - Have you collected any thread dumps from the broker once a consumer stops receiving messages? If so, what did they show? If not, could you? - Can you elaborate on what kind of and how many destinations you're using? You talk about some queues operating normally while other queues are having problems, but you also say that you're only using "one topic." - Is there a specific reason you're using transacted sessions?
Justin On Mon, Jan 2, 2023 at 12:17 PM Cezary Majchrzak < cezary.majchrza...@gmail.com> wrote: > Hello, > > We are observing strange communication problems with the ActiveMQ Artemis > broker in our system. When the problem occurs JmsListener stops receiving > further messages despite the fact that previously consuming worked > perfectly. The problem can occur on several queues but others at the same > time work properly. The Artemis management panel on the problematic queues > then indicates that deliveringCount > 0 and this value does not change. > Consumer count at this time is non-zero. Restarting the broker or message > consuming services does not always help. Sometimes messages are consumed > for a short time after which the problem reappears. We noticed that this > happens only when sending large messages (size of about 250 KB, Artemis > saves them with a size twice as large due to encoding). Problematic queues > process large and small messages or only large messages. Queues that work > properly process only small messages. At the same time, the problem does > not occur with every sending of large messages. We use message grouping, > assigning each message a UUID at the beginning of processing, which is then > used as a group identifier. We wonder if the large number of such groups > (sometimes even several million new messages per day) can have a > significant impact on memory consumption. > > > > *Artemis configuration* > > - Single instance of ActiveMQ Artemis broker (configured for > master-slave operation, but only one instance is enabled). > > - The broker is running on AlmaLinux 8.4 OS. > > - Artemis version is 2.27.1 (updated from version 2.22.0 where the > problem also occurred). > > - The broker.xml configuration file is attached. > > - One topic (omitting DLQ and ExpiryQueue) for which queues are > created with appropriate filters. > > *Application side configuration* > > - Spring Boot version 2.6.13 with spring-boot-starter-artemis. > > - Subscriptions configured as durable and shared. > > - Sessions are transacted. > > *What have we tried to solve the issue* > > - JmsListener used a container with dynamic scaling of the number > of consumers, while caching of consumers was enabled. We thought that this > might pose a problem for a broker trying to deliver messages to consumers > that no longer existed. We disabled caching of consumers and set > maxMessagePerTask property, unfortunately this did not solve the problem. > > - We tried changing Spring Boot's CachingConnectionFactory to > JmsPoolConnectionFactory from lib > https://github.com/messaginghub/pooled-jms, but again the problem was not > solved. > > - We took thread dumps in the services to make sure that the > processing doesn't get stuck when executing business logic and interacting > with external services. All threads of type JmsListenerEndpointContainer > are in TIMED_WAITING state and the stacktrace indicates that they are > waiting for messages from the broker in the receive method of class > org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl. > > - Updated the broker version to the latest 2.27.1, but the same > problem still occurs. > > - We tried changing the parameters of the acceptor in the > broker.xml file, such as: amqpMinLargeMessageSize (despite changing this > parameter, messages in the broker continue to be seen as large, despite the > smaller size than declared), remotingThreads and directDeliver. No apparent > effect on broker performance. > > - TCP dumps of the network traffic between the broker and the > services consuming the messages show that the network communication is > established and some data is sent from the broker. > > - We have changed the broker settings related to memory. > Previously, the host had 32GB of RAM and the Artemis process was configured > with the JVM -Xms and -Xmx parameters equal to 26GB and the global-max-size > parameter set by default. We noticed that during a heavy load of large > messages, in addition to the problem of not consuming messages, the host > would sometimes reset itself through out of out of memory errors. For this > reason, we increased the amount of RAM available to the host to 64GB and > set the -Xms and -Xmx parameters to 50G, and changed the global-max-size to > 10G as recommended by > https://activemq.apache.org/components/artemis/documentation/latest/perf-tuning.html. > The broker seemed to work more stably (one day processed about 3 million > large messages without any problems), unfortunately after about a week of > operation the problem of not consuming messages returned. I've attached > below graphs of memory consumption during one such problem. I have numbered > on them the consecutive times when we restarted the broker (coinciding with > high GC time and high committed memory value). During the first three > reboots, consuming resumed only for a moment, then stopped again. After the > fourth reboot, consuming started working properly and all the messages came > off the queues. > > > [image: memory_dump_1.png] > > > [image: memory_dump_2.png] > > > Similar symptoms have been described here > <https://stackoverflow.com/questions/74792977/no-data-being-sent-to-consumers-even-though-connection-and-session-are-created> > but the proposed solutions do not seem to apply to us. Please provide ideas > on how to solve the problem. > > Many thanks, > Cezary Majchrzak >
