> How much memory does each queue require of the broker? I just ran a quick test with a clean, default instance of 2.27.1 where I forced GC, inspected the heap size, created 1,000 durable anycast queues (which require 1,000 addresses as well), forced GC, then inspected the heap size again. When the test started the broker was using around 24MB of heap. After the test finished it was using around 560MB of heap. That means each address+queue is taking about 550KB of heap (i.e. (560-24)*1024/1000).
> What is the scanning overhead and when does it really become an issue? If we have a mostly-empty queue and handling 100/second requests is that going to be OK? If we have a queue with 100 messages backed up and 1000/second does that start to tip over the edge and become too slow? I can't really answer these questions as it's based on your specific use-case and performance goals. Justin On Thu, Dec 22, 2022 at 7:49 AM John Lilley <john.lil...@redpointglobal.com.invalid> wrote: > Mark, > > Yeah that's kind of where I'm coming from. We've seen that a large number > of queues consumes a lot of broker memory, and we've hit OOM a few times > due to 100s to reply-to temporary queues. We've since fixed the reply-to > queue count by having our RPC receivers multiplex a single reply-to queue. > However, we now face a similar tradeoff when it comes to our "job" RPC > queues, which are fairly low traffic (each job might get 5/second > messages). It would also be nice to multiplex all job RPC comms through a > single queue, because then if a job crashes we can have a "last chance" > service take over handling of all those jobs' requests and return errors. > We currently do so but we end up with a "last chance" service potentially > keeping 100s of queues active for a while "just in case" someone tries to > communicate with a crashed job. And that chews up broker memory, forcing us > to reduce the "last chance" responder time window. > > So scalability and throughput are not the only things driving this > decision. We also want to reduce the queue count to limit memory pressure > on the broker, and make it easier to cover for crashed jobs. > > I would really like know: > -- How much memory does each queue require of the broker? > -- What is the scanning overhead and when does it really become an issue? > If we have a mostly-empty queue and handling 100/second requests is that > going to be OK? If we have a queue with 100 messages backed up and > 1000/second does that start to tip over the edge and become too slow? > > Thanks > John > > > [image: rg] <https://www.redpointglobal.com/> > > John Lilley > > Data Management Chief Architect, Redpoint Global Inc. > > 888 Worcester Street, Suite 200 Wellesley, MA 02482 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > -----Original Message----- > From: Mark Johnson <spike.john...@gmail.com> > Sent: Thursday, December 22, 2022 5:35 AM > To: users@activemq.apache.org > Subject: Re: Message Selectors vs multiple queues > > *** [Caution] This email is from an external source. Please use caution > responding, opening attachments or clicking embedded links. *** > > Hi Justin, > > Is there a rule of thumb for the limit of Y? > > Just wondering about the realistic upper limit on the number of queues. > > Thanks > > On Wed, 21 Dec 2022 at 15:21, Justin Bertram <jbert...@apache.org> wrote: > > > Generally speaking I would avoid use-cases involving a JMS queue + > > selectors when possible for the following reasons: > > > > - The queue has to be scanned over and over for messages which match > > the selectors. The deeper the queue gets the more scanning is > > required. This adds up over time. > > - A single queue with X traffic will almost certainly not be as fast > > as Y queues with X/Y traffic on each. Put simplistically, more > > concurrency equals more performance both in terms of throughput and > scalability. > > > > > > Justin > > > > On Tue, Dec 20, 2022 at 5:26 PM John Lilley > > <john.lil...@redpointglobal.com.invalid> wrote: > > > >> Greetings! > >> > >> > >> > >> We have a design wherein every “job” that runs offers up an RPC > >> service via a unique queue name. Which means that if 100 jobs are > >> running, there are 100 queues. I’ve read that, in theory, all jobs > >> could share a single queue and use Message Selectors (in JMS) to > >> receive only messages that they should service. Can anyone help me > >> understand the tradeoffs? Would shared queue tend to use less broker > >> memory? Would the shared queue performance start to degrade under load? > >> > >> > >> > >> Thanks, > >> > >> John > >> > >> > >> > >> [image: rg] > >> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglob > >> al.com%2f&c=E,1,GWRpQJge7Z3JyOqH7F7YViXtCc3aUUWoOcyrofUIjoN0S_pkWLJjw > >> lLvrSeL-eorBWmJHJMwzHTmimlWsOH2WKdMQwd3GYf1io4GYNQV4yuXD9C8xKzFtQ,,&t > >> ypo=1> > >> > >> John Lilley > >> > >> Data Management Chief Architect, Redpoint Global Inc. > >> > >> 888 Worcester Street, Suite 200 Wellesley, MA 02482 > >> > >> *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > >> > >> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > >> confidential and is intended solely for the use of the individual(s) > >> to whom it is addressed. If you believe you received this e-mail in > >> error, please notify the sender immediately, delete the e-mail from > >> your computer and do not copy, print or disclose it to anyone else. > >> If you properly received this e-mail as a customer, partner or vendor > >> of Redpoint, you should maintain its contents in confidence subject > >> to the terms and conditions of your agreement(s) with Redpoint. > >> > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. >
