I am following up on this some more. ActiveMQ Classic has the capability to perform Idempotency built in, it just needs to be configured. For normal ActiveMQ usage I have not had any issues with duplicates. I am also able to enable the MemoryIdempotent server and do not have any issues.
The issue(s) I am experiencing relates to when ActiveMQ invokes the internal Camel components and performs a set of functions/routes/etc against messages in a queue. If Camel picks the message up and perform a set of actions against it and there is a failure of some type there is a potential, I have experienced this, where Camel will process the message again. This results in a duplicate message being created and a potential issue for an application/service. This would not be man issue if one broker or a master-slave broker was being used but I am using ActiveMQ as master-slave and network of brokers combined (clustering). Camel provides multiple options for Idempotency, Idempotent Consumer :: Apache Camel<https://camel.apache.org/components/4.10.x/eips/idempotentConsumer-eip.html> From the research I have conducted using an in-memory DB appears to be the best solution/option. The issue I am running into is that while I can figure out how to implement Idempotency within ActiveMQ and Camel for file based, the built-in in-memory based and a few others I am not able to find any infornmation on using Apache Cassandra for this and how it should be implemented. I have reas over the information multiple times and looked at other sources but I am struggling with how this should be mset/configured within the Camel route. Jason ________________________________ From: Matt Pavlovich Sent: Friday, August 30, 2024 12:02 PM To: users@activemq.apache.org Subject: Re: ActiveMQ Idempotentcy with Apache Cassandra CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Jason- ActiveMQ has several layers of audits built in to prevent duplicate messages on a per-producer, per-broker, and on a per-consumer basis. These do not go above the messaging application layer by design. This approach is appropriate for a messaging system where the goal is to reach queue size of zero (ie all messages processed) while being able to support scenarios such as failover and retry. Scenarios where you are looking to de-dupe messages based on _content_ or a _header value_ require a higher-level application component or integration. I suggest looking into the inbox and outbox patterns. When these are implemented, you are able to safely relax guarantees at the ActiveMQ layer and achieve insanely high throughput. Thanks, Matt Pavlovich > On Aug 29, 2024, at 7:06 AM, Jason Jackson > <jason.jack...@itechag.com.INVALID> wrote: > > I have run into an issue where I am occasionally receiving duplicate messages > when moving messages between ActiveMQ and a third party product using the > embedded Camel featured within ActiveMQ. > > According to the Apache web site this can happen during a failure of some > time. > > The way to resolve this issue is to use Idempotent consumer and check for > duplicate messages. > > I am using a network of brokers and need to do Idempotency through the entire > cluster. > > I am looking at using Apache Cassandra for this. I can get Cassandra up and > running but I am not able to find any clean/clear documentation that shows or > explains how to enable this for ActiveMQ/Camel routes. > > Does anyone have any links or information o how to perform this? > > > > Jason --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@activemq.apache.org For additional commands, e-mail: users-h...@activemq.apache.org For further information, visit: https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcontact&data=05%7C02%7Cjason.jackson%40itechag.com%7C011050248d9a41bc6b7008dcc90d2a94%7C07e5f1b9902a4d9f974c04601319bfec%7C0%7C0%7C638606305630649721%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=rGIYQLPXEboOXz5wNgdBL3uKWH4h7vcb1HH0NMVublg%3D&reserved=0<https://activemq.apache.org/contact>
