Hi list,
Currently we're seeing some strange behaviour from ActiveMQ 5.0 (release
version).
In our set-up we have one ActiveMQ instance with four queues on it and
all communication with AMQ is via the Stomp-interface. Although we
cleaned the activemq.xml a bit, we basically left all settings to their
respective defaults. We did change the Xmx-parameter in the bin-file to
2048M though since the below described problems are much worse with only
the default 512M.
We have five apache/php-webservers that connect to AMQ and produce one
or two message on every page request. They use the php-implementation of
Stomp and about 70% of those messages are persistent.
In peak moments, we do about 70-100 pageviews/second (and thus as many
connections to AMQ), which boils down to per-queue peaks of somewhere in
the 50 messages/second area.
Each queue has only one consumer. Those consumers throttle their
consumptionspeed based on the systemload, but normally they consume each
message as it enters a queue.
The connection/subscription from the consumers to ActiveMQ is opened at
start and then kept open for the duration of the program.
Under normal conditions there is at most one message in a queue waiting
to be consumed, but we've also seen it handle a (growing) queue of 100k
messages with ease.
We're noticing decreasing performance and increasing memory usage over
time. So two days ago I started logging the CC:PrintGCDetails and made a
graph of the free new/total heap before and after each young-gc (which
also shows the full gc's):
http://achelois.tweakers.net/~acm/tnet/activemq-gc.png
As you can see the new-part of the heap (green lines) isn't really
moving much, but the total heap (orange/red lines) keeps growing during
the day. What you also can see in the graph is the two nights AMQ was
running, and where no increase happened (from 20k to 40k seconds and
100k to 120k). Our nights are, obviously, much quieter than our days.
I've tried the Concurrent GC, but reverted to the Parallel GC since the
first one couldn't keep up when the heap grew over 700MB. AMQ is running
on a dual processor opteron with 8GB of memory and several 15k-rpm scsi
disks in a raid 5 for its storage.
I'm clueless as to what this increase in memory consumption may cause
and/or how to prevent it.
Any ideas?
Best regards,
Arjen van der Meijden
Tweakers.net
