ActiveMQ Artemis uses Netty for just about everything involving the network
and Netty uses direct memory.
Do you have any JVM on/off heap memory monitoring? If not I'd recommend
enabling the JVM and Netty [1] metrics, e.g.:
<metrics>
<jvm-memory>true</jvm-memory>
<netty-pool>true</netty-pool>
<plugin class-name="..."/>
</metrics>
You can, for example, scrape these with Prometheus [2], graph the usage,
and then correlate that with when you hit OOME.
I'm not aware of any specific memory recommendations other than to
benchmark, monitor, and tune for your specific use-case. There are many
different types of use-cases with widely varying memory usage patterns as
well as a wealth of tuning options.
Justin
[1]
https://activemq.apache.org/components/artemis/documentation/latest/metrics.html#optional-metrics
[2] https://github.com/jbertram/artemis-prometheus-metrics-plugin
On Wed, Jan 8, 2025 at 9:31 AM Vilius Šumskas <vilius.sums...@rivile.lt>
wrote:
> Thank you for your reply.
>
> VMs do not run anything else, just standard OpenSSH server and Google
> Cloud Monitoring Agent. I assume that these processes and the OS should fix
> into remaining 1.4GB. Usually these services consume just kilobytes of
> memory.
>
> JVM app in question is Artemis broker itself which is a standalone
> cluster. We run our JVM apps on Kubernetes cluster and connect to Artemis
> cluster externally.
>
> Usually of-the-shelf software running on JVM have their own
> recommendations. For example Jenkins
> https://docs.cloudbees.com/docs/cloudbees-ci/latest/jvm-troubleshooting/#_heap_size,
> or Sonatype Nexus Repository
> https://help.sonatype.com/en/nexus-repository-memory-overview.html .
> Hence my question regarding how to properly calculate Artemis memory sizes.
>
> The main question is where these 0.6GB is coming from if I specifically
> set Artemis heap size to 2G? Is this somekind of direct memory usage in the
> broker? I don't believe that this could be just JVM overhead.
>
> --
> Vilius
>
> -----Original Message-----
> From: David Bennion <dav...@gmx.com.INVALID>
> Sent: Wednesday, January 8, 2025 4:50 PM
> To: users@activemq.apache.org
> Subject: Re: avoiding oom-killer on Artemis
>
> OOM Killer happens when Linux determines it doesn't have enough memory to
> keep going, so in an act of self preservation it kills a large memory
> consuming process rather than crash.
>
> So in your case we can see from the OOM Killer log that in your case you
> had 2627224kB resident which I think is closer to 2.6GB. That leaves 1.4GB
> for the OS and all other applications.
>
> The follow up question is then: is there anything else running on the box
> other than your JVM? The total memory for all apps and the OS must fit
> within the 4GB max. So even small apps that use any memory can accumulate
> and cause Linux to perform OOM killer.
>
> Are you certain there is no memory leak in your JVM app? Does it take
> some time before you get OOM killed? A memory leak would certainly push
> your consumption on the OS to where garbage collection can't occur.
> Unfortunately, Linux will simply kill the app rather than ask Java to
> garbage collect. If there is no memory leak, I wonder if garbage collection
> settings could be modified to be more aggressive?
>
> Can the JVM app survive with less memory and still be performant? If you
> were to set the app to use 1.5 GB or 1GB what happens? Can you watch this
> problem occur on a console? If you run a "top" on that machine sorted by
> memory do you see other memory consumers? Are those expected?
>
> Regards,
> David
>
>
>
> On Jan 8, 2025, 12:31 AM, at 12:31 AM, "Vilius Šumskas" <
> vilius.sums...@rivile.lt> wrote:
> >Hi,
> >
> >I‘m trying to configure our Artemis cluster nodes in such a way that it
> >won‘t get oom-killed by Linux kernel, but instead paging would occur.
> >Most of the times it works, however after considerable periods of
> >paging sometimes we get:
> >
> >Jan 07 19:02:11 activemq-artemis-node-1 kernel:
> >oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allow
> >ed=0,global_oom,task_memcg=/system.slice/activemq.service,task=java,pid
> >=4086,uid=1001 Jan 07 19:02:11 activemq-artemis-node-1 kernel: Out of
> >memory: Killed process 4086 (java) total-vm:4939656kB,
> >anon-rss:2627224kB, file-rss:0kB, shmem-rss:0kB, UID:1001
> >pgtables:5588kB oom_score_adj:0
> >
> >Full oom-killer dump https://p.defau.lt/?HayAcFR8RRDrVZZAlav4IA
> >
> >Artemis nodes are using standard JVM options coming from official
> >distribution: -Xms512M -Xmx2G . They run in dedicated virtual hosts
> >which have 4GB assigned. Nothing else runs on those boxes.
> >global-max-size is not set so occupies 1G.
> >
> >Just wondering, how JVM heap sizes should be configured in relation to
> >available RAM? I found just this vague description
> >https://activemq.apache.org/components/artemis/documentation/latest/per
> >f-tuning.html#tuning-the-vm but nothing concrete. There is also a
> >question of JVM Direct Memory and other memory sectors which are not
> >controlled by heap settings. Does Artemis use those and what is
> >recommended ratios for them?
> >
> >--
> > Best Regards,
> >
> > Vilius Šumskas
> > Rivile
> > IT manager
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org
> For additional commands, e-mail: users-h...@activemq.apache.org
> For further information, visit: https://activemq.apache.org/contact
>
>
