Just copied the broker configuration file.
It's just a classic configuration and the only change we made is
1. The broker has persistent enabled because we have a few durable messages.
2. But for most of the messages, we don't want them go to the Temp Store, so
we used the memoryCursor.
3. To prevent memory overflow, we set the memoryLimit on each queue to be
200MB.
4. We allocate 9GB for the broker JVM and most of our traffic should be in
one queue, which shouldn't take more than 200MB. We also proved from the
heapdump that this Queue usage does not go over 200MB and there's hardly no
or very few content in other Queues or Topics. But the over all memory usage
still increase up to 5.89G due to a huge list under MemoryUsage.callbacks
(over 200 million items).
A few more coverage on the structure we have:
We have a broker network as follows
Producer -> broker1 -- n:1 --> *broker2* -> Consumer.
The problem we found is on *broker2* during the test on Consumer restart.
The restart takes around 30 minutes. So during that time, the 200mb queue
limit is reached on broker2. And there're a lot more pending messages
accumulated on broker1 side. (broker1 has TempStore enabled so it can cache
more much more data).
Once the Consumer is back line, it starting to consume messages from
broker2. The MemoyUsage spike is sported at this point of time.
One more thing to stress is. 99.9% of the data are transmitted through one
queue and it has memory limit by 200MB on broker1. The maximum throughput on
this Queue is
15000 (msg / sec) x 1.6KB = 24MB / sec.
No matter from the data volume or the total number of messages, it can't
explain why MemoyUsage class consumed 5.66G and generate 220 Million
callback listeners within 5 mins. In some test, it directly drain the
broker's memory and cause the broker hang!
We'd like to know:
1. why it happens? A Queue usage monitoring class consumed much more memory
than the Queue itself. Is
2. this an ActiveMQ defect or our configuration mistake.
3. Is there a way to disable this MemoryUsage or avoid this spike from the
configuration?
Thanks a lot in advance!
-Yang
It doesn't make sense broker1
--
View this message in context:
http://activemq.2283324.n4.nabble.com/org-apache-activemq-usage-MemoryUsage-consumes-95-of-the-memory-tp4701250p4701285.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
