Off the top of my head I can't think of any reasons that message grouping necessarily wouldn't work with redistribution. Can you outline your exact use-case so I can test it myself. If you have a minimal, reproducible example (e.g. on GitHub or something) that would be ideal.
Justin On Mon, Apr 14, 2025 at 2:20 AM Luchi Mirko <mirko.lu...@siav.it> wrote: > > Hi Justin, > > for what concerns connection router, I'll take a better look at it (I > quickly read the chapter some times ago but haven't made any serious > thoughts about it yet). > > For what concerns message redistribution, I had already experimented with > that, but it seems to me that when message grouping is enabled, > redistribution is not working. I thought it was an issue related to my > configuration, but using the same configuration and repeating the test > without using message grouping (i.e. as soon as I don't send messages with > group id) redistribution works like a charm. > Is message redistribution supposed to work even when message grouping is > used? > > > Mirko > ________________________________ > From: Justin Bertram <jbert...@apache.org> > Sent: Friday, April 11, 2025 21:30 > To: users@activemq.apache.org <users@activemq.apache.org> > Subject: Re: Clustered Grouping: how to make sure that each queue on the > cluster has at least one consumer registered > > > Is there a way to specify some kind of policy by which client should > preferably first connect to queues without consumers... > > There is no policy to change where _consumers_ get connected, but you can > use a connection router [1] in combination with your cluster to distribute > _connections_ in a particular way. > > In any event, if you have redistribution [2] configured then messages > should not build-up on any specific node as long as there is a consumer on > the subscription on at least one node in the cluster so this problem is > likely moot. > > I realize you're just exploring "what can be done and what can't," but the > devil is in the details with this kind of thing, and as you add layers of > complexity it becomes increasingly difficult to say what's possible and > what isn't. Furthermore, nothing scales indefinitely so the more concrete > you can be about your requirements the easier it will be to provide clear > answers. Unfortunately vague questions tend to get vague answers. > > > Justin > > [1] > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fconnection-routers.html%23connection-routers&e=acf6d3bc&h=1c8f6aa2&f=y&p=y > [2] > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fclusters.html%23message-redistribution&e=acf6d3bc&h=e5f6ee21&f=y&p=y > > > On Fri, Apr 11, 2025 at 12:55 PM Luchi Mirko <mirko.lu...@siav.it> wrote: > > > Hi Justin. > > > > 1)I read that a single broker is often more than enough to handle massive > > number of messages, but I haven't executed any performance test yet. > > Since what we're doing is evaluating Artemis, we want to know what can be > > done and what can't. > > Besides we don't know right now what size out future customers may be, so > > we'd like to understand if clustering is feasible in case it was needed. > > I have already experimented HA configuration, but that's high > > availability, not load balancing. > > > > 2) Yes, I've read them and understood. But as I said we are performing an > > evaluation and comparison with alternatives, and clustering is usually > one > > of the parameters we evaluate. > > > > Could you please suggest an approach to address my requirement? > > Thanks. > > > > > > ________________________________ > > From: Justin Bertram <jbert...@apache.org> > > Sent: Friday, April 11, 2025 7:29:09 PM > > To: users@activemq.apache.org <users@activemq.apache.org> > > Subject: Re: Clustered Grouping: how to make sure that each queue on the > > cluster has at least one consumer registered > > > > Before we get into the details of your use-case I have a few questions... > > > > 1) Have you conducted benchmark tests and demonstrated conclusively that > > you cannot meet your performance goals with a single broker (or HA pair > of > > brokers)? If so, can you share any details about your testing and the > > results? > > 2) Have you read about and understood the performance considerations for > > clustering [1]? > > > > > > Justin > > > > [1] > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fclusters.html%23performance-considerations&e=acf6d3bc&h=9c39a666&f=y&p=y > > > > On Fri, Apr 11, 2025 at 9:39 AM Luchi Mirko <mirko.lu...@siav.it> wrote: > > > > > Hi, > > > > > > we are planning to adopt Artemis in our software solution as substitute > > of > > > the current message broker (RabbitMQ). > > > We are especially interested in 2 features it offers: > > > > > > * > > > message grouping > > > * > > > automatic rebalancing of message groups. > > > > > > We need message grouping because our consumers must ensure that certain > > > groups of messages are processed serially. > > > We need rebalancing because we deploy our solution in an elastic > > > environment and we want to be able to scale the application > horizontally. > > > > > > Ideally, we would like to use a cluster of Artemis brokers for load > > > balancing, so we read the clustered grouping chapter of the > > documentation ( > > > > > > https://urlsand.esvalabs.com/?u=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fmessage-grouping.html%23clustered-grouping&e=acf6d3bc&h=2f3421ef&f=y&p=y > > ) > > > and we are aware of the potential pitfalls in this choice. > > > > > > To simplify let's say we have a single address to which we publish > > > messages (each message with groupId set), and multiple shared durable > > > subcriptions that load balance consumption of those messages. > > > Given that we will scale the application horizontally, we can assume > that > > > each node of the Artemis cluster will eventually have a queue bound to > > that > > > address with some groupIds sticked to it. > > > We know that we must make sure that each of these queues must have at > > > least one consumer attached, or we might find ourselves in a situation > > > where a message is routed (because of message grouping) to a queue that > > has > > > no consumers attached, and hence that message will never get a chance > to > > be > > > consumed. > > > > > > We can make sure (properly tuning autoscaling of our consumer pods) > that > > > the minimum number of consumers is always equals or greater than the > > number > > > of broker nodes (a.k.a. number of queues), but even doing so it might > > > happen that multiple consumers register on the same queue on the same > > > broker node, leaving therefore queues without consumers. > > > Is there a way to specify some kind of policy by which client should > > > preferably first connect to queues without consumers, so that consumers > > are > > > distributed evenly and we can guarantee that no queue will remain > without > > > consumer (given of course that we make sure to have at least a number > of > > > consumers equal to the broker nodes)? > > > > > > Thanks > > > > > > > > > > > >
