If you have a dedicated topicPartition for delayed messages, you can pause <https://static.javadoc.io/org.apache.kafka/kafka-clients/2.2.0/org/apache/kafka/clients/consumer/KafkaConsumer.html#pause-java.util.Collection-> that partition for 15 min to avoid blocking the polling thread.
On Thu, May 23, 2019 at 12:16 PM Adrien Ruffie <adrye...@gmail.com> wrote: > Thank a lot ! That's what I thought, out of the box ... but it's clearly > logic > I will take your solution in the consumer, but I will refactor my Java code > part, because, it's a spring message listener with annotation. So, I > think, I couldn't do it simply with spring. > > thank again, > > best regards > Adrian > > Le mer. 22 mai 2019 à 23:23, Jonathan Santilli <jonathansanti...@gmail.com > > > a écrit : > > > Maybe you could: > > > > 1.- Disable auto.commit on the consumer side. > > 2.- Consume messages, one by one in each poll. > > 3.- Check the record timestamp > > a.- If the timestamp is >= desired window (time slot), process it and > > commit offset > > b.- If the timestamp is < desired window (time slot), do not process > it > > and do not commit offset, in the next poll the consumer will get the > > message again. > > > > Hope that helps, > > -- > > Jonathan > > > > > > > > > > On Wed, May 22, 2019 at 9:25 PM Peter Bukowinski <pmb...@gmail.com> > wrote: > > > > > I’d suggest using separate topics for messages that require delay and > > ones > > > that do not. If you are limited to a single topic, then I’d use some > > other > > > metadata to differentiate messages that require delayed processing from > > > ones that do not. If you do not want to block the polling thread, > you’ll > > > need to route messages into a buffer of some sort to process them > > > asynchronously. > > > > > > — > > > Peter > > > > > > > On May 22, 2019, at 1:10 PM, Pavel Molchanov < > > > pavel.molcha...@infodesk.com> wrote: > > > > > > > > This solution will block receiving polling thread for 15 minutes. Not > > > good. > > > > > > > > What should we do if a topic has messages that should be processed > > > > immediately and delayed messages at the same time? > > > > > > > > *Pavel Molchanov* > > > > > > > > > > > > > > > > On Wed, May 22, 2019 at 2:41 PM Peter Bukowinski <pmb...@gmail.com> > > > wrote: > > > > > > > >> There is no out-of-the-box way to tell a consumer to not consume an > > > offset > > > >> until it is x minutes old. Your best bet is encode the creation time > > > into > > > >> the message themselves and add some processing logic into your > > consumer. > > > >> Let’s assume your topic has a single partition or your partitions > are > > > keyed > > > >> to guarantee message order. Your consumer could work like this in > > > >> pseudo-code: > > > >> > > > >> consumer loop: > > > >> consume message > > > >> if (current time - message.timestamp) >= 15 minutes > > > >> process message > > > >> else > > > >> sleep 15 minutes - (current time - message.timestamp) > > > >> process message > > > >> > > > >> Since the messages enter the topic in the order they were published, > > > >> pausing on the current offset should never cause a bottleneck on the > > > later > > > >> messages. If you fall behind, the greater than or equal to logic > will > > > >> prevent your consumer from pausing until it has caught up to your > > > desired > > > >> delay. > > > >> > > > >> This is a simplified scenario that may or may not map to your > > production > > > >> use case, though. > > > >> > > > >> — > > > >> Peter > > > >> > > > >> > > > >>> On May 22, 2019, at 11:12 AM, Pavel Molchanov < > > > >> pavel.molcha...@infodesk.com> wrote: > > > >>> > > > >>> Andrien, > > > >>> > > > >>> Thank you for asking this question! I have the same problem and > > wanted > > > to > > > >>> ask the same question. I hope that someone will answer soon. > > > >>> > > > >>> *Pavel Molchanov* > > > >>> > > > >>> > > > >>> > > > >>> On Wed, May 22, 2019 at 9:54 AM Adrien Ruffie <adrye...@gmail.com> > > > >> wrote: > > > >>> > > > >>>> Hello all, > > > >>>> > > > >>>> I have a specific need and I don't know if a generic solution > exist > > > ... > > > >>>> maybe you can enlighten me > > > >>>> > > > >>>> I need to delay each sended message about 15 mins. > > > >>>> Example > > > >>>> Message with offset 1 created at 2:41PM by the producer and > received > > > by > > > >> the > > > >>>> consumer at 2:56PM > > > >>>> Message with offset 2 created at 2:46PM by the producer and > received > > > by > > > >> the > > > >>>> consumer at 3:01PM > > > >>>> Message with offset 3 created at 2:46PM by the producer and > received > > > by > > > >> the > > > >>>> consumer at 3:01PM > > > >>>> Message with offset 4 created at 3:01PM by the producer and > received > > > by > > > >> the > > > >>>> consumer at 3:16PM > > > >>>> and so forth ... > > > >>>> > > > >>>> any option, mechanism, producer/consumer implementations already > > > exist ? > > > >>>> > > > >>>> Thank a lot and best regards > > > >>>> > > > >>>> Adrian > > > >>>> > > > >> > > > >> > > > > > > > > > > -- > > Santilli Jonathan > > >
