Re: Delayed Queue

[Jagan Ranganathan]

Thanks Aniket for your suggestion. 

I am new to Kafka, so wonder if there is something like a TTL and deadletter 
exchange similar to RabbitMQ, which kind of does the job for me. That kind of 
simplifies things.

Regards,
Jagan

---- On Sun, 23 Feb 2014 15:44:46 +0530 Aniket 
Bhatnagar<aniket.bhatna...@gmail.com> wrote ---- 


Alright. Here is an approach that I can think of. Create a topic for each 
delay and produce message to the appropriate topic. Each produced message 
should have a timestamp at which it was pushed to the queue. At the 
consumer side, fetch a message from a partition and compare the message 
timestamp with system's timestamp to see if enough time has passed for you 
to process the message. If enough time has passed, process the message and 
commit the message's offset otherwise make sure you do not commit the 
offset. 
 
You may have to use low level consumer API to achieve this, however double 
check if you can use high level consumer API to achieve the same (as I am 
not an expert of the consumer APIs). 
 
Also, this assumes the size of your delay set is small as it is 
recommended to only have couple of thousand partitions. 
 On 23 Feb 2014 15:15, "Jagan Ranganathan" <ja...@zohocorp.com> wrote: 
 
> Hi Aniket, 
> 
> 
> I am looking at "Finite set of delays" and messages will use one on the 
> defined set. 
> 
> 
> Regards, 
> Jagan 
> 
> ---- On Sun, 23 Feb 2014 15:13:02 +0530 Aniket 
> Bhatnagar<aniket.bhatna...@gmail.com> wrote ---- 
> 
> 
> Hi Jagan 
> 
> Are you expecting each message to have the same delay? Or is there a 
finite 
> set of delays? Or messages can arbitrarily any delay? 
> 
> Thanks, 
> Aniket 
> On 23 Feb 2014 14:13, "Jagan Ranganathan" 
<ja...@zohocorp.com> 
> wrote: 
> 
> > Dear All, 
> > 
> > I am looking for a Delayed Queue implementation for my 
distributed 
> system 
> > requirement. Would like to know if I can achieve the same using 
Kafka. 
> > 
> > 
> > Regards, 
> > Jagan 
> > 
> > 
> 
> 
> 
>