> Do message group IDs expire? No. They do not expire. However, you can deal with this is several ways:
1) You can "close" the group from the producer [1]. 2) You can use group "buckets" [2] to limit memory consumption. 3) You can administratively reset the groups using the resetAllGroups() [3] or resetGroup(String) [4] management operation on the relevant QueueControl. Justin [1] https://activemq.apache.org/components/artemis/documentation/latest/message-grouping.html#closing-a-message-group [2] https://activemq.apache.org/components/artemis/documentation/latest/message-grouping.html#group-buckets [3] https://activemq.apache.org/components/artemis/documentation/javadocs/javadoc-latest/org/apache/activemq/artemis/api/core/management/QueueControl.html#resetAllGroups() [4] https://activemq.apache.org/components/artemis/documentation/javadocs/javadoc-latest/org/apache/activemq/artemis/api/core/management/QueueControl.html#resetGroup(java.lang.String) On Mon, Dec 11, 2023 at 8:08 AM John Lilley <john.lil...@redpointglobal.com.invalid> wrote: > Justin, > > > > Thanks again for your attention to this. I have verified many times now > that message groups work in our test harness -- > > When I enable message groups the test succeeds, and when I disable message > groups the test fails. > > > > Your concern about message groups was: “Even in the case of message > grouping, the messages are not directed to a specific consumer as much as > they are just directed at _just one consumer (i.e. any consumer) at a time_ > since the goal is just for the messages to be processed in order.” > > > > In our case, all three types of RPC request messages for a given task > (startLongThing, isLongThingFinished, and getLongThingResult) get the same > message group ID. Logically the RPC message stream would look like: > > > > REQUEST: {"request": "start_long_thing", "message_group":"123456"} > > RESPONSE: {"task_id":"1"} > > REQUEST: {"request": "is_long_thing_finished", "task_id":"1", > "message_group":"123456"} > > RESPONSE: {"finished":"false"} > > REQUEST: {"request": "is_long_thing_finished", "task_id":"1", > "message_group":"123456"} > > RESPONSE: {"finished":"false"} > > ... > > REQUEST: {"request": "is_long_thing_finished", "task_id":"1", > "message_group":"123456"} > > RESPONSE: {"finished":"true"} > > REQUEST: {"request": "get_long_thing_result", "task_id":"1", > "message_group":"123456"} > > RESPONSE: {"result":{...}} > > > > All of the "is_long_thing_finished" requests are polling at a rate of, > let’s say, once per second. > > > > As far as our application is concerned, the important thing is that all of > these messages with the same message group Id are deliver to the same > consumer. It doesn’t matter which one. I believe that your description of > how message groups work does indeed match our needs. > > > > However, there are some things that concern me about the behavior of > message groups which are not documented, and I’m hoping you or someone on > the group can fill in the blanks for me. > > > > My fundamental question is: *Do message group IDs expire?* > > Every task that we create gets a unique message group ID. Over the course > of time, the cumulative set of unique message group IDs is unbounded. > > > > This leads to the first question – if the broker tracks message group IDs > “forever”, will this cause a memory leak in the broker as the tracking of > these IDs accumulates? > > > > The flip side of that question: if the broker expires message group IDs > and eventually forgets about them, how much time is a message group ID > available before it expires? Does the tracking of the message group > continue to get renewed so long as it is used periodically? > > > > Thanks > > John > > > > > > > [image: rg] <https://www.redpointglobal.com/> > > John Lilley > > Data Management Chief Architect, Redpoint Global Inc. > > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > *From:* Justin Bertram <jbert...@apache.org> > *Sent:* Saturday, December 9, 2023 10:49 PM > *To:* users@activemq.apache.org > *Subject:* Re: Is it possible to route message to a specific consumer? > > > > **** [Caution] This email is from an external source. Please use caution > responding, opening attachments or clicking embedded links. **** > > > > I wouldn't expect message grouping to help given the information about > your use-case that you've provided so far. I outlined the reasons for this > in my previous email. However, if you say that it's working for you then > there's almost certainly something about your use-case that I don't > understand. > > > > Typically in request/response use-cases the requester specifies a > "correlation ID" to allow it to keep track of the exchange of messages. > Also, in cases with multiple brokers it's typical to cluster those brokers > together so that messages can be shared transparently across the different > brokers or perhaps to use connection routers [1] to "shard" applications to > particular brokers or sets of brokers so that the applications can get > reconnected back to the proper broker(s) in case they get disconnected > assuming that the brokers contain application-related state information. I > assume you're using something like this already, but if not it may be worth > looking into. > > > > > > Justin > > > > [1] > https://activemq.apache.org/components/artemis/documentation/latest/connection-routers.html#connection-routers > > > > > > Justin > > > > On Fri, Dec 8, 2023 at 5:26 PM John Lilley < > john.lil...@redpointglobal.com.invalid> wrote: > > Hi Justin, > > > > Thanks for the response! > > > > Our use case is something like this. Suppose we have a nice RPC client > wrapped around message queues and their services. Some methods take a long > time, and we want to report status along the way, so we’ve arranged for > such methods to return a “task ID”: > > > > String taskID = client.startLongThing(); > > while (!client.isLongThingFinished(taskID)) { wait, or print progress… } > > client.getLongThingResult(taskID); > > > > The problem we hit is that the long-running tasks are not stateless. They > start, execute and finish within the same server process. So the calls to > startLongThing(), isLongThingFinished(), and getLongThingResult() must all > be serviced by the same process. If we have multiple HA instances of a > service running in a Kubernetes cluster, the isLongThingFinished() requests > will be handle by a random server that may not know about the task. > > > > Do you think that the “message group” attribute is a reasonable approach? > In JMS: > > message.setStringProperty("JMSXGroupID", "grouplabel"); > > > > I’ve implemented and tested this and it does seem to work. Do you know of > any caveats or cautions? > > > > Thanks > > John > > > > > > > > [image: rg] > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,MiJvWl9nWaaSsiJBFyc7CGAQMRDxVHUdDSfPqakKAOm_SablTwk4IQKKszbYlDIh5j4aSqYxuECDLsDRtkQVkoSrqFbw7DnV0Oi9x8uMfXhoDTfu1yRjKFO_DpM,&typo=1> > > *John Lilley * > > *Data Management Chief Architect, Redpoint Global Inc. * > > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > *From:* Justin Bertram <jbert...@apache.org> > *Sent:* Thursday, December 7, 2023 10:56 PM > *To:* users@activemq.apache.org > *Subject:* Re: Is it possible to route message to a specific consumer? > > > > **** [Caution] This email is from an external source. Please use caution > responding, opening attachments or clicking embedded links. **** > > > > > Is there any kind of “consumer affinity” that can be attached to a > message, which directs the broker to select a specific consumer? > > > > I may be forgetting something, but I believe the answer to this question > is no. The broker is broadly designed to support decoupled, asynchronous > messaging where consumers and producers know essentially nothing about each > other and the broker just receives messages and makes them available for > consumption. Message consumption is fundamentally consumer-driven. The > broker itself doesn't choose where the messages go except in some limited > sense in particular use-cases. Even in the case of message grouping, the > messages are not directed to a specific consumer as much as they are just > directed at _just one consumer (i.e. any consumer) at a time_ since the > goal is just for the messages to be processed in order. > > > > You can add specific properties to specific messages and then JMS > consumers can use selectors to either ignore or choose those messages for > consumption and in this way you can ensure that certain consumers get > certain messages, but both the producer and consumer need to know about the > properties. Furthermore, if all the applications use a single queue then > they can start interfering with each other. For example, if one application > fails to consume its messages then messages will begin accumulating and may > prevent other applications from getting their messages since queues are by > nature first-in-first-out data structures. > > > > I'd need to know more about the specifics of your use-case before I could > make further recommendations. > > > > > > Justin > > > > On Thu, Dec 7, 2023 at 1:26 PM John Lilley < > john.lil...@redpointglobal.com.invalid> wrote: > > Oooh, maybe I answered my own question: > > > https://activemq.apache.org/components/artemis/documentation/1.0.0/message-grouping.html > > This *looks* like what I’m after. Any advice around its use? > > > > John > > > > > > > > [image: rg] > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,RLCz7uLdUVWU-4PdO0mzCFR75gph7wgFAJpvnoG5yg55RbBDZxZfAUVZs-ERB9VGZHVUZK6w6h0r3FI5bjkHA50Th_z9pS5huRB5AZIUhxN5nTIQlTbVa2sLLA,,&typo=1> > > *John Lilley * > > *Data Management Chief Architect, Redpoint Global Inc. * > > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > *From:* John Lilley <john.lil...@redpointglobal.com.INVALID> > *Sent:* Thursday, December 7, 2023 11:58 AM > *To:* users@activemq.apache.org > *Subject:* Is it possible to route message to a specific consumer? > > > > **** [Caution] This email is from an external source. Please use caution > responding, opening attachments or clicking embedded links. **** > > > > Greetings, > > > > We are in the process of transforming our Java-based services into > Kubernetes-based microservice swarms with HA topology. So every service > will have multiple instances. The good news is, AMQ supports this > splendidly, because the message broker is very good about load-balancing > between consumers and handling failed consumers. > > > > However, we have a few cases of “long running tasks” that a single service > instance is responsible for. In those cases, we need to be able to route a > request to a specific consumer instance. We could use specially-named > queues for each consumer instance, but this seems awkward and hopefully not > necessary. > > > > Is there any kind of “consumer affinity” that can be attached to a > message, which directs the broker to select a specific consumer? > > If not, other ideas are certainly welcome. > > > > We are using OpenJDK 17 > > > > This is our maven dependency > > <dependency> > <groupId>org.apache.activemq</groupId> > <artifactId>artemis-jms-client-all</artifactId> > <version>2.30.0</version> > </dependency> > > > > > > Thanks > > John > > > > > > *John Lilley > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>* > > *Data Management Chief Architect, Redpoint Global Inc. > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>* > > 34 Washington Street, Suite 205 Wellesley Hills, MA 02481 > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1> > > *M: +1 7209385761 | john.lil...@redpointglobal.com > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>* > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1> > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1> > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. >
