Can you create a minimal producer via the OpenWire protocol in Java or another language of your choice, to determine if your Camel producer is slow because it's OpenWire or because it's Camel? I suspect you'll find that OpenWire is the culprit, not Camel, but let's confirm that.
All of these numbers sound tiny compared to what the ActiveMQ product is capable of (though I don't have any insight into how Amazon has configured the brokers, nor into any code customizations they might have made). If you run multiple minimal producers in parallel, does throughput increase linearly? Also, you say you're testing with small payloads; are they small enough that you might be running into the Nagle algorithm on your TCP sockets? If you use larger (e.g. 1KB) payloads, what does that do to your throughput on a single producer? Tim On Thu, Aug 22, 2019, 2:54 AM James Green <james.mk.gr...@gmail.com> wrote: > Hi all, > > I've been busy shifting an existing workload into AWS recently, and a load > test shows a serious performance drop when sending to ActiveMQ which I > could use some advice on. > > Quick architecture summary: We send requests via a webserver that are > forwarded as messages to a queue. A backend receives these messages and > forwards them onward to another queue. Spring Boot with Camel powers the > show within Docker containers. Messages are persistent. > > Story so far: > > Tests show this first queue builds rapidly with pending messages yet > monitoring of our existing production environment shows no such backlog. > > Our existing production environment has everything in a single DC so it's > super low latency. Our AWS environment uses Fargate with AmazonMQ. I > understand send latency will be higher and AmazonMQ will store the messages > across three AZs. > > So I launched a small EC2 instance to run some comparison tests: > > Receiving via a Camel route is super quick. This is not a problem. > Sending via a minimal Camel route is super slow. 14 messages per second. We > appear to be doing at least 20-30 per second in production but it's enough > of a difference. > Sending via PHP with stomp-php setting both persistence on and receipt > headers on is substantially faster than sending via Camel. 55 messages per > second. > Tests have been with 10K small payloads. > > At this point I'm thinking that both Camel and PHP should be sending with > the same properties - synchronously and with persistence. The messages on > the queue are flagged persistent when viewed by the web console. > > Can anyone provide further suggestions to try? > > Thanks, > > James >
