Hi Justin, for what concerns connection router, I'll take a better look at it (I quickly read the chapter some times ago but haven't made any serious thoughts about it yet).
For what concerns message redistribution, I had already experimented with that, but it seems to me that when message grouping is enabled, redistribution is not working. I thought it was an issue related to my configuration, but using the same configuration and repeating the test without using message grouping (i.e. as soon as I don't send messages with group id) redistribution works like a charm. Is message redistribution supposed to work even when message grouping is used? Mirko ________________________________ From: Justin Bertram <jbert...@apache.org> Sent: Friday, April 11, 2025 21:30 To: users@activemq.apache.org <users@activemq.apache.org> Subject: Re: Clustered Grouping: how to make sure that each queue on the cluster has at least one consumer registered > Is there a way to specify some kind of policy by which client should preferably first connect to queues without consumers... There is no policy to change where _consumers_ get connected, but you can use a connection router [1] in combination with your cluster to distribute _connections_ in a particular way. In any event, if you have redistribution [2] configured then messages should not build-up on any specific node as long as there is a consumer on the subscription on at least one node in the cluster so this problem is likely moot. I realize you're just exploring "what can be done and what can't," but the devil is in the details with this kind of thing, and as you add layers of complexity it becomes increasingly difficult to say what's possible and what isn't. Furthermore, nothing scales indefinitely so the more concrete you can be about your requirements the easier it will be to provide clear answers. Unfortunately vague questions tend to get vague answers. Justin [1] https://urlsand.esvalabs.com/?u=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fconnection-routers.html%23connection-routers&e=acf6d3bc&h=1c8f6aa2&f=y&p=y [2] https://urlsand.esvalabs.com/?u=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fclusters.html%23message-redistribution&e=acf6d3bc&h=e5f6ee21&f=y&p=y On Fri, Apr 11, 2025 at 12:55 PM Luchi Mirko <mirko.lu...@siav.it> wrote: > Hi Justin. > > 1)I read that a single broker is often more than enough to handle massive > number of messages, but I haven't executed any performance test yet. > Since what we're doing is evaluating Artemis, we want to know what can be > done and what can't. > Besides we don't know right now what size out future customers may be, so > we'd like to understand if clustering is feasible in case it was needed. > I have already experimented HA configuration, but that's high > availability, not load balancing. > > 2) Yes, I've read them and understood. But as I said we are performing an > evaluation and comparison with alternatives, and clustering is usually one > of the parameters we evaluate. > > Could you please suggest an approach to address my requirement? > Thanks. > > > ________________________________ > From: Justin Bertram <jbert...@apache.org> > Sent: Friday, April 11, 2025 7:29:09 PM > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: Clustered Grouping: how to make sure that each queue on the > cluster has at least one consumer registered > > Before we get into the details of your use-case I have a few questions... > > 1) Have you conducted benchmark tests and demonstrated conclusively that > you cannot meet your performance goals with a single broker (or HA pair of > brokers)? If so, can you share any details about your testing and the > results? > 2) Have you read about and understood the performance considerations for > clustering [1]? > > > Justin > > [1] > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fclusters.html%23performance-considerations&e=acf6d3bc&h=9c39a666&f=y&p=y > > On Fri, Apr 11, 2025 at 9:39 AM Luchi Mirko <mirko.lu...@siav.it> wrote: > > > Hi, > > > > we are planning to adopt Artemis in our software solution as substitute > of > > the current message broker (RabbitMQ). > > We are especially interested in 2 features it offers: > > > > * > > message grouping > > * > > automatic rebalancing of message groups. > > > > We need message grouping because our consumers must ensure that certain > > groups of messages are processed serially. > > We need rebalancing because we deploy our solution in an elastic > > environment and we want to be able to scale the application horizontally. > > > > Ideally, we would like to use a cluster of Artemis brokers for load > > balancing, so we read the clustered grouping chapter of the > documentation ( > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fmessage-grouping.html%23clustered-grouping&e=acf6d3bc&h=2f3421ef&f=y&p=y > ) > > and we are aware of the potential pitfalls in this choice. > > > > To simplify let's say we have a single address to which we publish > > messages (each message with groupId set), and multiple shared durable > > subcriptions that load balance consumption of those messages. > > Given that we will scale the application horizontally, we can assume that > > each node of the Artemis cluster will eventually have a queue bound to > that > > address with some groupIds sticked to it. > > We know that we must make sure that each of these queues must have at > > least one consumer attached, or we might find ourselves in a situation > > where a message is routed (because of message grouping) to a queue that > has > > no consumers attached, and hence that message will never get a chance to > be > > consumed. > > > > We can make sure (properly tuning autoscaling of our consumer pods) that > > the minimum number of consumers is always equals or greater than the > number > > of broker nodes (a.k.a. number of queues), but even doing so it might > > happen that multiple consumers register on the same queue on the same > > broker node, leaving therefore queues without consumers. > > Is there a way to specify some kind of policy by which client should > > preferably first connect to queues without consumers, so that consumers > are > > distributed evenly and we can guarantee that no queue will remain without > > consumer (given of course that we make sure to have at least a number of > > consumers equal to the broker nodes)? > > > > Thanks > > > > > > >
