It has been a while but i've been a bit busy.
I've restarted the broker a couple of times in the last 2 weeks and the
situation did not change.
The broker also restarted on its own a couple of times due to address mem usage
reaching 100%.
I've removed this part of the config yesterday and restarted the broker:
<broker-plugins>
<broker-plugin
class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
<property key="LOG_ALL_EVENTS" value="true"/>
<property key="LOG_CONNECTION_EVENTS" value="true"/>
<property key="LOG_SESSION_EVENTS" value="true"/>
<property key="LOG_CONSUMER_EVENTS" value="true"/>
<property key="LOG_DELIVERING_EVENTS" value="true"/>
<property key="LOG_SENDING_EVENTS" value="true"/>
<property key="LOG_INTERNAL_EVENTS" value="true"/>
</broker-plugin>
</broker-plugins>
The mem usage now sits below 10mb, traffic remains the same.
Check the data from zabbix from the last 2 days - there's some missing data in
between, disregard that ......
mem usage:
https://i.imgur.com/zYRF3tT.png
traffic (msgs added):
https://i.imgur.com/bm0pURD.png
I strongly believe the above config causes a mem leak.
[1658227403484]
Gašper Čefarin
Systems Engineer
Ferrarska ulica 14, 6000 Koper – Slovenia
M: +386 51 281 905
W: http://www.actual-it.si
________________________________
From: Gašper Čefarin <[email protected]>
Sent: 11 September 2025 10:11
To: [email protected]
Subject: Re: Large address size for address with no messages in queues
I understand you now - im running this on ubuntu 24, and looking at ps or htop
gives me the following (calculated from percentages) numbers for mem usage:
ps: 1373 MiB
htop: 1350 MiB
while console address mem usage is at: 1055MiB
Numbers make sense to me, since the broker mem usage should be higher than
address mem.
Secondary instance meanwhile reports:
ps: 145 MiB
htop: 133 MiB
console address mem: 0.00 MiB
When I refer to console address mem, I'm looking at Artemis/status page.
Zabbix item for address mem usage that I referred to in the previous mail:
jmx["org.apache.activemq.artemis:broker=\"{#JMXOBJ}\"",AddressMemoryUsage]
There's no indication that the values for mem usage in MiB in the console are
wrong.
There is however a bug for calculating percentages and I believe it comes from
the broker, not the console.
Right now, both the web console and this zabbix item:
jmx["org.apache.activemq.artemis:broker=\"{#JMXOBJ}\"",AddressMemoryUsagePercentage]
report 140% address mem usage. I'll add this to jira.
-------
I also found this under JMX/java.lang/Memory (Still web console):
HeapMemoryUsage
{"init":134217728,"committed":1167065088,"max":1572864000,"used":880245424}
NonHeapMemoryUsage
{"init":7667712,"committed":151453696,"max":-1,"used":119638448}
ObjectName {"objectName":"java.lang:type=Memory"}
ObjectPendingFinalizationCount 0
Verbose false
Zabbix items like
jmx["java.lang:type=Memory",HeapMemoryUsage.committed]
report the same numbers.
I'll try to reproduce and hopefully get a heap dump on OOM.
________________________________
From: Justin Bertram <[email protected]>
Sent: 10 September 2025 17:19:40
To: [email protected]
Subject: Re: Large address size for address with no messages in queues
To sporočilo izvira izven naše organizacije. Bodite pozorni pri vsebini in
odpiranju povezav ali prilog.
> The "problem" is that I have the instance consuming 1000mb of memory
while I have about 300 messages waiting to be consumed...
It's not clear to me if there is an actual increase in heap memory being
consumed (which might indicate a real memory leak) or if the broker is
simply over-reporting address memory usage (which indicates a simpler
accounting error). Do you have any insight into which of these is
happening? What exact metrics are you looking at via the web console and
via Zabbix?
At this point I'd say you should work on a minimal reproducer. Once that's
in hand the use-case and operating environment will be clear.
Justin
On Wed, Sep 10, 2025 at 3:22 AM Gašper Čefarin <[email protected]>
wrote:
> The "problem" is that I have the instance consuming 1000mb of memory while
> I have about 300 messages waiting to be consumed, and the biggest ones are
> around 30kb in size.
>
>
> I was inspecting via web console. I still haven't restarted the broker,
> the address memory size is up to 1000MB now (according to the web console,
> and zabbix monitoring.) Persistent sizes for queues are around 300kb max
> according to zabbix.
>
>
>
> I did restart the broker after the changes. The journal-buffer.size is at
> default - I don't have it in my settings.
>
> This is a "test" instance and i'm just testing different configurations -
> the idea was to keep the majority of messages in memory - and some of the
> messages that go through our broker are larger than 100kb. I would expect
> the memory to go up if the messages are still waiting in the queues - but
> vast majority of them are consumed right away (the ones that aren't are
> usually somewhere in a dead letter queue, or a low-traffic queue ...)
>
>
> I'm not sure how to reproduce, but I can try. I didn't want to restart the
> broker in case there's any additional info I can get while the instance is
> still running.
>
> There's one more setting that I have enabled even before
> minLargeMessageSize, not sure when. Maybe this is the culprit and I didn't
> notice increased mem usage before.
>
>
>
> <!-- Uncomment the following if you want to use the Standard
> LoggingActiveMQServerPlugin pluging to log in events -->
> <broker-plugins>
> <broker-plugin
> class-name="org.apache.activemq.artemis.core.server.plugin.impl.LoggingActiveMQServerPlugin">
> <property key="LOG_ALL_EVENTS" value="true"/>
> <property key="LOG_CONNECTION_EVENTS" value="true"/>
> <property key="LOG_SESSION_EVENTS" value="true"/>
> <property key="LOG_CONSUMER_EVENTS" value="true"/>
> <property key="LOG_DELIVERING_EVENTS" value="true"/>
> <property key="LOG_SENDING_EVENTS" value="true"/>
> <property key="LOG_INTERNAL_EVENTS" value="true"/>
> </broker-plugin>
> </broker-plugins>
>
> If you think I can get more data from the running instance, i'll wait, if
> not I can restart and try to reproduce. I also configured
> HeapDumpOnOutOfMemoryError.
>
>
>
>
>
> ________________________________
> From: Justin Bertram <[email protected]>
> Sent: 09 September 2025 22:52
> To: [email protected]
> Subject: Re: Large address size for address with no messages in queues
>
>
> To sporočilo izvira izven naše organizacije. Bodite pozorni pri vsebini in
> odpiranju povezav ali prilog.
>
>
>
>
> When you say "the address consumes ~400MB of memory" are you inspecting
> this from the web console or some other way?
>
> Did you restart the broker after making your configuration changes? Did you
> happen to adjust the journal-buffer-size?
>
> Was there a specific reason you increased the minLargeMessageSize? That
> could certainly lead to much higher memory utilization.
>
> Do you have a way to reproduce the issue?
>
> At this point I'm not sure what the "problem" is or if there really is a
> problem.
>
>
> Justin
>
> On Fri, Sep 5, 2025 at 7:53 AM Gašper Čefarin <[email protected]
> >
> wrote:
>
> > Hi,
> > i'm using activemq artemis v2.41.0 -
> >
> > I have an address that has 1 empty (no messages) queue. Even so, the
> > address consumes ~400MB of memory (queue1).
> >
> > The second largest address consumes around 21MB of memory. It also only
> > has 1, currently empty queue (queue2).
> >
> > These 2 queues have the highest "messages added/acked" count: 646640 and
> > 648931 respectively (notice that address with less msgs has bigger
> address
> > size).
> >
> > If at all important: all of the messages received in queue1 should go to
> > queue2 (using camel routes and a couple of custom processors - which
> should
> > leave the message body unchanged)
> >
> > So far, ive only noticed address memory rising if there were queues
> > waiting to be acknowledged. What's going on - is this a possible memory
> > leak? Do you need any more info?
> >
> > Thanks
> >
> >
> >
>
