Hi,
I use ActiveMQ 5.9.0 on FreeBSD with java 1.7.0_25-b15 and native
leveldb (default settings).
Four servers, fully meshed network of brokers, only two get effective load.
I publish from a small number of servers (around 30) to a virtual topic,
then consume the aggregated messages from one STOMP client.
The average message size is around 250 bytes, the incoming message rate
is around 8k+6k messages per second on the two servers.
What I see is the flow towards the consumer periodically pauses (maybe
the incoming flow too, but I don't monitor that).
Measuring three of those pauses gave 34.01463485, 34.03087783 and
53.02782011 seconds, during which no messages were delivered to the
client at all. The first two are pretty suspicious.
Digging into it a little, "top" on the ActiveMQ server shows that one
thread is active during the pauses, and it's mostly in pipe read state. WTF.
Doing a ktrace (strace-like FreeBSD tool) shows that it spawns multiple
"ln" commands (yikes) which mess with the leveldb files (I guess it's
the compaction job).
It seems others have performance issues with leveldb too (like
http://symas.com/mdb/hyperdex/, maybe you should also switch to LMDB :)
), so instead of playing with its tunable knobs (I've already failed
with them, see my previous mail about kahadb being faster than leveldb),
I've just switched to kahadb (also default settings) to check whether
this causes the delays.
Yes, leveldb causes the long pauses in message deliveries. I'm running
two hours since I've switched, and I haven't yet had a second with zero
delivered messages, and the overall average rate of message deliveries
per seconds is also smoothed out, no fluctuation.
Are you aware of this issue? Skimming through the list it seems there
may be others hit by this, but I couldn't yet find an e-mail which would
state that leveldb is bad for performance, you should a) avoid it, b)
set these knobs to these, c) wait for the next revision, we'll fix this.
Thanks,
