Mark,

Yeah that's kind of where I'm coming from. We've seen that a large number of 
queues consumes a lot of broker memory, and we've hit OOM a few times due to 
100s to reply-to temporary queues. We've since fixed the reply-to queue count 
by having our RPC receivers multiplex a single reply-to queue. However, we now 
face a similar tradeoff when it comes to our "job" RPC queues, which are fairly 
low traffic (each job might get 5/second messages). It would also be nice to 
multiplex all job RPC comms through a single queue, because then if a job 
crashes we can have a "last chance" service take over handling of all those 
jobs' requests and return errors. We currently do so but we end up with a "last 
chance" service potentially keeping 100s of queues active for a while "just in 
case" someone tries to communicate with a crashed job. And that chews up broker 
memory, forcing us to reduce the "last chance" responder time window.

So scalability and throughput are not the only things driving this decision. We 
also want to reduce the queue count to limit memory pressure on the broker, and 
make it easier to cover for crashed jobs.

I would really like know:
-- How much memory does each queue require of the broker?
-- What is the scanning overhead and when does it really become an issue? If we 
have a mostly-empty queue and handling 100/second requests is that going to be 
OK? If we have a queue with 100 messages backed up and 1000/second does that 
start to tip over the edge and become too slow?

Thanks
John



[rg] <https://www.redpointglobal.com/>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

888 Worcester Street, Suite 200 Wellesley, MA 02482

M: +1 7209385761<tel:+1%207209385761> | 
john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com>

-----Original Message-----
From: Mark Johnson <spike.john...@gmail.com<mailto:spike.john...@gmail.com>>
Sent: Thursday, December 22, 2022 5:35 AM
To: users@activemq.apache.org<mailto:users@activemq.apache.org>
Subject: Re: Message Selectors vs multiple queues

*** [Caution] This email is from an external source. Please use caution 
responding, opening attachments or clicking embedded links. ***

Hi Justin,

Is there a rule of thumb for the limit of Y?

Just wondering about the realistic upper limit on the number of queues.

Thanks

On Wed, 21 Dec 2022 at 15:21, Justin Bertram 
<jbert...@apache.org<mailto:jbert...@apache.org>> wrote:

> Generally speaking I would avoid use-cases involving a JMS queue +
> selectors when possible for the following reasons:
>
> - The queue has to be scanned over and over for messages which match
> the selectors. The deeper the queue gets the more scanning is
> required. This adds up over time.
> - A single queue with X traffic will almost certainly not be as fast
> as Y queues with X/Y traffic on each. Put simplistically, more
> concurrency equals more performance both in terms of throughput and 
> scalability.
>
>
> Justin
>
> On Tue, Dec 20, 2022 at 5:26 PM John Lilley
> <john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>>
>  wrote:
>
>> Greetings!
>>
>>
>>
>> We have a design wherein every “job” that runs offers up an RPC
>> service via a unique queue name. Which means that if 100 jobs are
>> running, there are 100 queues. I’ve read that, in theory, all jobs
>> could share a single queue and use Message Selectors (in JMS) to
>> receive only messages that they should service. Can anyone help me
>> understand the tradeoffs? Would shared queue tend to use less broker
>> memory? Would the shared queue performance start to degrade under load?
>>
>>
>>
>> Thanks,
>>
>> John
>>
>>
>>
>> [image: rg]
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglob
>> al.com%2f&c=E,1,GWRpQJge7Z3JyOqH7F7YViXtCc3aUUWoOcyrofUIjoN0S_pkWLJjw
>> lLvrSeL-eorBWmJHJMwzHTmimlWsOH2WKdMQwd3GYf1io4GYNQV4yuXD9C8xKzFtQ,,&t
>> ypo=1>
>>
>> John Lilley
>>
>> Data Management Chief Architect, Redpoint Global Inc.
>>
>> 888 Worcester Street, Suite 200 Wellesley, MA 02482
>>
>> *M: *+1 7209385761 <+1%207209385761> | 
>> john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com>
>>
>> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is
>> confidential and is intended solely for the use of the individual(s)
>> to whom it is addressed. If you believe you received this e-mail in
>> error, please notify the sender immediately, delete the e-mail from
>> your computer and do not copy, print or disclose it to anyone else.
>> If you properly received this e-mail as a customer, partner or vendor
>> of Redpoint, you should maintain its contents in confidence subject
>> to the terms and conditions of your agreement(s) with Redpoint.
>>
>

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential 
and is intended solely for the use of the individual(s) to whom it is 
addressed. If you believe you received this e-mail in error, please notify the 
sender immediately, delete the e-mail from your computer and do not copy, print 
or disclose it to anyone else. If you properly received this e-mail as a 
customer, partner or vendor of Redpoint, you should maintain its contents in 
confidence subject to the terms and conditions of your agreement(s) with 
Redpoint.

Reply via email to