Hi, Following-up as I've run more tests.
My minimal producer suffered the same bug as our main application: we had spring boot activemq thread pooling turned on as a property, but the library (referenced in the main docs) was not included. Looking back at my rather sparse notes at the time I activated this my sends of 10K messages went from taking 11m47s to between 2m56s - 3m31s which is a marked improvement. To my chagrin, this has made little different to our real-world application, and so I have modified my minimal producer to be capable of sending to the main application via it's queues. Allow me to elaborate at this point as it's important to understand what I'm looking at... The messages follow a small path through a series of queues as they are processed. Queue A -> B -> C. If my minimal producer sends to Queue C (skipping A and B) I'm able to produce at 49/s which is "quick enough". If my minimal producer sends to Queue B (skipping A) I'm able to produce at 28/s - 38/s which is variable but most of the tests reached 38/s. If my minimal producer sends to Queue A I'm able to produce at 28/s - 42/s - again variable. Now Queues A and B are consumed by separate Camel routes inside the same application. Queue C is entirely separate. Looking at throughput graphs of the consumption of Queue C, when first going through (A,B) for 10K messages, then going through (B), I can see (A,B) is twice as slow. I'm left wondering if there's contention somehow within the application consuming from (A,B) that is only showing up during load testing on AWS, I was not expecting it would be 2x slower unless the producer thread is shared - you might imagine a thread pool was solve that! At this point I have ensured that there are 4 instances of each application and they can happily deal with about 50 messages per second across the queues with persistence on. I am uncertain whether I should be expecting more. If anyone has insights on why the two routes within the same application appear contended and indeed on whether overall throughput should be a lot higher I'd love to hear it. James On Thu, 22 Aug 2019 at 14:02, Tim Bain <tb...@alumni.duke.edu> wrote: > Can you create a minimal producer via the OpenWire protocol in Java or > another language of your choice, to determine if your Camel producer is > slow because it's OpenWire or because it's Camel? I suspect you'll find > that OpenWire is the culprit, not Camel, but let's confirm that. > > All of these numbers sound tiny compared to what the ActiveMQ product is > capable of (though I don't have any insight into how Amazon has configured > the brokers, nor into any code customizations they might have made). If you > run multiple minimal producers in parallel, does throughput increase > linearly? > > Also, you say you're testing with small payloads; are they small enough > that you might be running into the Nagle algorithm on your TCP sockets? If > you use larger (e.g. 1KB) payloads, what does that do to your throughput on > a single producer? > > Tim > > On Thu, Aug 22, 2019, 2:54 AM James Green <james.mk.gr...@gmail.com> > wrote: > > > Hi all, > > > > I've been busy shifting an existing workload into AWS recently, and a > load > > test shows a serious performance drop when sending to ActiveMQ which I > > could use some advice on. > > > > Quick architecture summary: We send requests via a webserver that are > > forwarded as messages to a queue. A backend receives these messages and > > forwards them onward to another queue. Spring Boot with Camel powers the > > show within Docker containers. Messages are persistent. > > > > Story so far: > > > > Tests show this first queue builds rapidly with pending messages yet > > monitoring of our existing production environment shows no such backlog. > > > > Our existing production environment has everything in a single DC so it's > > super low latency. Our AWS environment uses Fargate with AmazonMQ. I > > understand send latency will be higher and AmazonMQ will store the > messages > > across three AZs. > > > > So I launched a small EC2 instance to run some comparison tests: > > > > Receiving via a Camel route is super quick. This is not a problem. > > Sending via a minimal Camel route is super slow. 14 messages per second. > We > > appear to be doing at least 20-30 per second in production but it's > enough > > of a difference. > > Sending via PHP with stomp-php setting both persistence on and receipt > > headers on is substantially faster than sending via Camel. 55 messages > per > > second. > > Tests have been with 10K small payloads. > > > > At this point I'm thinking that both Camel and PHP should be sending with > > the same properties - synchronously and with persistence. The messages on > > the queue are flagged persistent when viewed by the web console. > > > > Can anyone provide further suggestions to try? > > > > Thanks, > > > > James > > >
