Thank a lot ! That's what I thought, out of the box ... but it's clearly logic I will take your solution in the consumer, but I will refactor my Java code part, because, it's a spring message listener with annotation. So, I think, I couldn't do it simply with spring.
thank again, best regards Adrian Le mer. 22 mai 2019 à 23:23, Jonathan Santilli <jonathansanti...@gmail.com> a écrit : > Maybe you could: > > 1.- Disable auto.commit on the consumer side. > 2.- Consume messages, one by one in each poll. > 3.- Check the record timestamp > a.- If the timestamp is >= desired window (time slot), process it and > commit offset > b.- If the timestamp is < desired window (time slot), do not process it > and do not commit offset, in the next poll the consumer will get the > message again. > > Hope that helps, > -- > Jonathan > > > > > On Wed, May 22, 2019 at 9:25 PM Peter Bukowinski <pmb...@gmail.com> wrote: > > > I’d suggest using separate topics for messages that require delay and > ones > > that do not. If you are limited to a single topic, then I’d use some > other > > metadata to differentiate messages that require delayed processing from > > ones that do not. If you do not want to block the polling thread, you’ll > > need to route messages into a buffer of some sort to process them > > asynchronously. > > > > — > > Peter > > > > > On May 22, 2019, at 1:10 PM, Pavel Molchanov < > > pavel.molcha...@infodesk.com> wrote: > > > > > > This solution will block receiving polling thread for 15 minutes. Not > > good. > > > > > > What should we do if a topic has messages that should be processed > > > immediately and delayed messages at the same time? > > > > > > *Pavel Molchanov* > > > > > > > > > > > > On Wed, May 22, 2019 at 2:41 PM Peter Bukowinski <pmb...@gmail.com> > > wrote: > > > > > >> There is no out-of-the-box way to tell a consumer to not consume an > > offset > > >> until it is x minutes old. Your best bet is encode the creation time > > into > > >> the message themselves and add some processing logic into your > consumer. > > >> Let’s assume your topic has a single partition or your partitions are > > keyed > > >> to guarantee message order. Your consumer could work like this in > > >> pseudo-code: > > >> > > >> consumer loop: > > >> consume message > > >> if (current time - message.timestamp) >= 15 minutes > > >> process message > > >> else > > >> sleep 15 minutes - (current time - message.timestamp) > > >> process message > > >> > > >> Since the messages enter the topic in the order they were published, > > >> pausing on the current offset should never cause a bottleneck on the > > later > > >> messages. If you fall behind, the greater than or equal to logic will > > >> prevent your consumer from pausing until it has caught up to your > > desired > > >> delay. > > >> > > >> This is a simplified scenario that may or may not map to your > production > > >> use case, though. > > >> > > >> — > > >> Peter > > >> > > >> > > >>> On May 22, 2019, at 11:12 AM, Pavel Molchanov < > > >> pavel.molcha...@infodesk.com> wrote: > > >>> > > >>> Andrien, > > >>> > > >>> Thank you for asking this question! I have the same problem and > wanted > > to > > >>> ask the same question. I hope that someone will answer soon. > > >>> > > >>> *Pavel Molchanov* > > >>> > > >>> > > >>> > > >>> On Wed, May 22, 2019 at 9:54 AM Adrien Ruffie <adrye...@gmail.com> > > >> wrote: > > >>> > > >>>> Hello all, > > >>>> > > >>>> I have a specific need and I don't know if a generic solution exist > > ... > > >>>> maybe you can enlighten me > > >>>> > > >>>> I need to delay each sended message about 15 mins. > > >>>> Example > > >>>> Message with offset 1 created at 2:41PM by the producer and received > > by > > >> the > > >>>> consumer at 2:56PM > > >>>> Message with offset 2 created at 2:46PM by the producer and received > > by > > >> the > > >>>> consumer at 3:01PM > > >>>> Message with offset 3 created at 2:46PM by the producer and received > > by > > >> the > > >>>> consumer at 3:01PM > > >>>> Message with offset 4 created at 3:01PM by the producer and received > > by > > >> the > > >>>> consumer at 3:16PM > > >>>> and so forth ... > > >>>> > > >>>> any option, mechanism, producer/consumer implementations already > > exist ? > > >>>> > > >>>> Thank a lot and best regards > > >>>> > > >>>> Adrian > > >>>> > > >> > > >> > > > > > > -- > Santilli Jonathan >
