Hi JB
I was using a EBS (local) partition when I saw the horrendous I/O
throughput. During testing over the last hour or so, I've mounted an EFS
(for those who don't grok Amazon AWS, Elastic File System: basically an NFS
mount) target and symlinked the kahadb directory over to it.
My current KahaDB configuration is really nothing special:
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>
However, the good news is that initial testing using EFS has reduced the
load on the server substantially, and the other tasks - in particular, a
Camel route which takes a JSON list, converts it to individual messages and
converts each back to a JSON - now run within 5ms, whereas before they were
taking upwards of 1200ms per message.
I am building a cluster in development, and I'll look at upgrading to
5.15.9 or .10. Using a JDBC store might be a better fit for me, as I have
plenty of spare capacity on a PostgreSQL server. Is this likely to scale
up to a few hundred messages a second, or is KahaDB a better way to go?
Peter
On Tue, 22 Oct 2019 at 20:20, Jean-Baptiste Onofré <j...@nanthrax.net> wrote:
> Hi Peter,
>
> The most important is the I/O rate/throughput. I'm also using some
> brokers on EC2 (some using JDBC store, some using kahadb store).
>
> What's the filesystem ? EFS or "local" EC2 ?
>
> What's your current kahadb configuration in activemq.xml ?
>
> Just a note: 5.15.9 got major improvements on kahadb that could help.
>
> Regards
> JB
>
> On 22/10/2019 19:54, Peter Hicks wrote:
> > All,
> >
> > I have a feed of 110 messages/second of about 150 bytes each which I'm
> > routing through a default-settings ActiveMQ 5.15.8 server and sending on
> to
> > a topic. Everything works fine until I set up a durable subscription, at
> > which point iostat (Ubuntu 18.04LTS) reports about 300tps and about 2-3
> > megabytes a second of disk writes, which seems like an awful lot of the
> > message rate and size, and it's slowing down other processing on the
> > server. Is this normal and expected?
> >
> > The server is within Amazon EC2 and I can easily add an additional disk
> for
> > the KahaDB directory, but can anyone point me at a resource that will
> help
> > me reduce the I/O requirements of persisting all these messages to
> disk? I
> > am open to any suggestions.
> >
> >
> > Peter
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
OpenTrainTimes Ltd. registered in England and Wales, company no.
09504022.
Registered office: 13a Davenant Road, Upper Holloway, London N19
3NW
