Hi, I think you can try disable concurrentStoreAndDispatchQueues and rerun the tests.
Alan Diego On Wed, Aug 28, 2019 at 9:42 AM James Green <james.mk.gr...@gmail.com> wrote: > Hi, > > Following-up as I've run more tests. > > My minimal producer suffered the same bug as our main application: we had > spring boot activemq thread pooling turned on as a property, but the > library (referenced in the main docs) was not included. Looking back at my > rather sparse notes at the time I activated this my sends of 10K messages > went from taking 11m47s to between 2m56s - 3m31s which is a marked > improvement. > > To my chagrin, this has made little different to our real-world > application, and so I have modified my minimal producer to be capable of > sending to the main application via it's queues. > > Allow me to elaborate at this point as it's important to understand what > I'm looking at... > > The messages follow a small path through a series of queues as they are > processed. Queue A -> B -> C. > > If my minimal producer sends to Queue C (skipping A and B) I'm able to > produce at 49/s which is "quick enough". > If my minimal producer sends to Queue B (skipping A) I'm able to produce at > 28/s - 38/s which is variable but most of the tests reached 38/s. > If my minimal producer sends to Queue A I'm able to produce at 28/s - 42/s > - again variable. > > Now Queues A and B are consumed by separate Camel routes inside the same > application. Queue C is entirely separate. > > Looking at throughput graphs of the consumption of Queue C, when first > going through (A,B) for 10K messages, then going through (B), I can see > (A,B) is twice as slow. > > I'm left wondering if there's contention somehow within the application > consuming from (A,B) that is only showing up during load testing on AWS, I > was not expecting it would be 2x slower unless the producer thread is > shared - you might imagine a thread pool was solve that! > > At this point I have ensured that there are 4 instances of each application > and they can happily deal with about 50 messages per second across the > queues with persistence on. I am uncertain whether I should be expecting > more. > > If anyone has insights on why the two routes within the same application > appear contended and indeed on whether overall throughput should be a lot > higher I'd love to hear it. > > James > > > On Thu, 22 Aug 2019 at 14:02, Tim Bain <tb...@alumni.duke.edu> wrote: > > > Can you create a minimal producer via the OpenWire protocol in Java or > > another language of your choice, to determine if your Camel producer is > > slow because it's OpenWire or because it's Camel? I suspect you'll find > > that OpenWire is the culprit, not Camel, but let's confirm that. > > > > All of these numbers sound tiny compared to what the ActiveMQ product is > > capable of (though I don't have any insight into how Amazon has > configured > > the brokers, nor into any code customizations they might have made). If > you > > run multiple minimal producers in parallel, does throughput increase > > linearly? > > > > Also, you say you're testing with small payloads; are they small enough > > that you might be running into the Nagle algorithm on your TCP sockets? > If > > you use larger (e.g. 1KB) payloads, what does that do to your throughput > on > > a single producer? > > > > Tim > > > > On Thu, Aug 22, 2019, 2:54 AM James Green <james.mk.gr...@gmail.com> > > wrote: > > > > > Hi all, > > > > > > I've been busy shifting an existing workload into AWS recently, and a > > load > > > test shows a serious performance drop when sending to ActiveMQ which I > > > could use some advice on. > > > > > > Quick architecture summary: We send requests via a webserver that are > > > forwarded as messages to a queue. A backend receives these messages and > > > forwards them onward to another queue. Spring Boot with Camel powers > the > > > show within Docker containers. Messages are persistent. > > > > > > Story so far: > > > > > > Tests show this first queue builds rapidly with pending messages yet > > > monitoring of our existing production environment shows no such > backlog. > > > > > > Our existing production environment has everything in a single DC so > it's > > > super low latency. Our AWS environment uses Fargate with AmazonMQ. I > > > understand send latency will be higher and AmazonMQ will store the > > messages > > > across three AZs. > > > > > > So I launched a small EC2 instance to run some comparison tests: > > > > > > Receiving via a Camel route is super quick. This is not a problem. > > > Sending via a minimal Camel route is super slow. 14 messages per > second. > > We > > > appear to be doing at least 20-30 per second in production but it's > > enough > > > of a difference. > > > Sending via PHP with stomp-php setting both persistence on and receipt > > > headers on is substantially faster than sending via Camel. 55 messages > > per > > > second. > > > Tests have been with 10K small payloads. > > > > > > At this point I'm thinking that both Camel and PHP should be sending > with > > > the same properties - synchronously and with persistence. The messages > on > > > the queue are flagged persistent when viewed by the web console. > > > > > > Can anyone provide further suggestions to try? > > > > > > Thanks, > > > > > > James > > > > > >
