Hello,
Inside Wildfly 23.0.0 we are running ActiveMQ Artemis Message Broker
2.16.0 with JDBC persistence on SQLServer for ~25 message queues. In
some production environments we have moderate-to-high message volumes
and since processing can be relatively slow temporary message pileup is
not uncommon.
In one particular environment we are experiencing OutOfMemory issues
during startup. There are about 60K messages in 2 of the queues while
the message table contains over 350M records causing memory exhaustion
during startup. Running in a controlled environment with ~60G heap
startup succeeds and through JProfiler we observe that all message table
records are selected and appear to be collected in memory. After that
they are processed and discarded dropping memory usage down to a
fraction of its peak. Using the JBoss CLI to inspect the queues then
shows we have indeed 60K messages in 2 queues.
Inspecting the contents of the message table we see limited counts of
record types 13 (ADD_RECORD_TX) and 14 (UPDATE_RECORD_TX) roughly
equivalent to the 60K message count. All remaining records are type 11
(ADD_RECORD).
When we removed all type 11 records restart was fast with limited memory
load and we still see 60K messages in the 2 queues.
In the past we have observed similar numbers and startup problems in
other environments which lead us to truncate the AMQ persistence tables
to be able to restart the server without an OutOfMemoryException. But we
are looking for a way to prevent this situation from happening.
So my questions are:
- Is the large record count in the message table expected behavior?
- Is there anything we can/should do to limit the number of records in
the message table?
- Is removing all type 11 records a valid workaround? If no, what would
be the side-effect?
Thanks for your help!
Silvio
