Hi Jagan, unfortunately Kafka doesn't have the same TTL feature as you find in RabbitMQ. This is because Kafka and RabbitMQ have a fundamentally different design:
- RabbitMQ brokers individually track the status of each message (whether it has been acked by a consumer, when its TTL expires, etc) - Kafka simply appends a message to a partition, and then processes each partition sequentially. The Kafka broker doesn't keep track of the status of messages; it's up to consumers to keep track of which messages they've seen, based on their current offset in each partition. Kafka's design has the advantage that it scales to much higher message throughput, and the fact that messages are never re-ordered is a useful feature in some applications. But it means that something like a TTL (which inherently re-orders messages) is not easily done in Kafka, except through a construction like Aniket described. Martin On 23 Feb 2014, at 10:29, Jagan Ranganathan <ja...@zohocorp.com> wrote: > Thanks Aniket for your suggestion. > > I am new to Kafka, so wonder if there is something like a TTL and deadletter > exchange similar to RabbitMQ, which kind of does the job for me. That kind of > simplifies things. > > Regards, > Jagan > > ---- On Sun, 23 Feb 2014 15:44:46 +0530 Aniket > Bhatnagar<aniket.bhatna...@gmail.com> wrote ---- > > > Alright. Here is an approach that I can think of. Create a topic for each > delay and produce message to the appropriate topic. Each produced message > should have a timestamp at which it was pushed to the queue. At the > consumer side, fetch a message from a partition and compare the message > timestamp with system's timestamp to see if enough time has passed for you > to process the message. If enough time has passed, process the message and > commit the message's offset otherwise make sure you do not commit the > offset. > > You may have to use low level consumer API to achieve this, however double > check if you can use high level consumer API to achieve the same (as I am > not an expert of the consumer APIs). > > Also, this assumes the size of your delay set is small as it is > recommended to only have couple of thousand partitions. > On 23 Feb 2014 15:15, "Jagan Ranganathan" <ja...@zohocorp.com> wrote: > > > Hi Aniket, > > > > > > I am looking at "Finite set of delays" and messages will use one on the > > defined set. > > > > > > Regards, > > Jagan > > > > ---- On Sun, 23 Feb 2014 15:13:02 +0530 Aniket > > Bhatnagar&lt;aniket.bhatna...@gmail.com&gt; wrote ---- > > > > > > Hi Jagan > > > > Are you expecting each message to have the same delay? Or is there a > finite > > set of delays? Or messages can arbitrarily any delay? > > > > Thanks, > > Aniket > > On 23 Feb 2014 14:13, "Jagan Ranganathan" > &lt;ja...@zohocorp.com&gt; > > wrote: > > > > &gt; Dear All, > > &gt; > > &gt; I am looking for a Delayed Queue implementation for my > distributed > > system > > &gt; requirement. Would like to know if I can achieve the same using > Kafka. > > &gt; > > &gt; > > &gt; Regards, > > &gt; Jagan > > &gt; > > &gt; > > > > > > > > > > > > >
