Hi. I asked this question here
https://stackoverflow.com/questions/79671868/heap-memory-use-more-than-global-max-size-specified,
but it seems like the topic couldn`t be bound just to one question, so here
I am.
I`ve done some load tests to check the heap consumption in broker paging
mode and found out, that heap could be reduced by limiting the parameters
follows:
- global-max-size=200Mb
- page-size-bytes=1M
- max-read-page-bytes=2M
- prefetch-page-messages=1 (that`s ok for me, because I`m not going to
use multiple consumers of one queue)
Config with global-max-size=200Mb and other default values has the heap
around 1.4Gb.
Config with limited bytes (see above) has heap reduced to around 400Mb, and
that looks good.
Could you please confirm that I`m on the right path here? Are there any
other parameters critical to heap memory that I`ve missed?
Also in SO answer it was mentioned by Justin that the journal type won't
meaningfully change how much memory the broker uses.
But in tests I see that with same broker parameters (see above) for *large
messages* (280Kb) the NIO and JDBC journals show different heap memory
usage:
- NIO - around 400Mb
- JDBC - around 800Mb
Is it any JDBC-specific memory overhead that could cause higher heap usage?
--------------------------------
Producer client to load messages I`ve used:
> package org.acme.jms.producer;
>
> import java.nio.charset.StandardCharsets;
> import java.util.Set;
> import java.util.UUID;
> import java.util.concurrent.CountDownLatch;
> import java.util.concurrent.ExecutorService;
> import java.util.concurrent.Executors;
> import java.util.stream.Collectors;
> import java.util.stream.Stream;
>
> import javax.jms.BytesMessage;
> import javax.jms.DeliveryMode;
> import javax.jms.JMSContext;
> import javax.jms.JMSProducer;
> import javax.jms.Queue;
>
> import org.apache.qpid.jms.JmsConnectionFactory;
>
> public class MessageProducer
> {
> static final int CLIENT_COUNT = 100;
> static final int MESSAGE_COUNT = 200;
> static final int PRODUCE_THREADS = 24;
> static final String URL = "amqp://localhost:61616";
> static final String USER = "usr";
> static final String PWD = "pwd";
> static final String MSG_70KB = Stream.generate(() -> "bytes test
> msg").limit(5000).collect(Collectors.joining());
> static final String MSG_280KB = Stream.generate(() -> "bytes test
> msg").limit(20000).collect(Collectors.joining());
>
> public static void main(String... args)
> {
> try
> {
> Set<String> queues = Stream.generate(() ->
> UUID.randomUUID().toString())
> .limit(CLIENT_COUNT)
> .collect(Collectors.toSet());
>
> produceMessages(queues, MSG_70KB);
> //produceMessages(queues, MSG_280KB);
> }
> catch (Exception e)
> {
> throw new RuntimeException(e);
> }
> }
>
> private static void produceMessages(Set<String> queues, String msg)
> throws InterruptedException
> {
> CountDownLatch latch = new CountDownLatch(CLIENT_COUNT - 1);
> ExecutorService executorService =
> Executors.newFixedThreadPool(PRODUCE_THREADS);
>
> for (String queueName : queues)
> {
> executorService.execute(() ->
> {
> JmsConnectionFactory jmsConnectionFactory = new
> JmsConnectionFactory(USER, PWD, URL);
> try (JMSContext jmsContext =
> jmsConnectionFactory.createContext(JMSContext.AUTO_ACKNOWLEDGE))
> {
> jmsContext.start();
>
> Queue queue = jmsContext.createQueue(queueName);
> JMSProducer producer = jmsContext.createProducer();
> producer.setDeliveryMode(DeliveryMode.PERSISTENT);
>
> for (int j = 0; j < MESSAGE_COUNT; j++)
> {
> BytesMessage jmsMsg = jmsContext.createBytesMessage();
> jmsMsg.setJMSDeliveryMode(DeliveryMode.PERSISTENT);
>
> jmsMsg.writeBytes(getTestMessage(msg).getBytes(StandardCharsets.UTF_8));
>
> producer.send(queue, jmsMsg);
> }
> }
> catch (Exception e)
> {
> throw new RuntimeException("jms msg send failed", e);
> }
>
> latch.countDown();
> });
> }
> latch.await();
> executorService.shutdown();
> }
>
> private static String getTestMessage(String msg)
> {
> return UUID.randomUUID() + msg;
> }
> }
>
>
Regards, Sergey.
