Hi Florian, It's actually OK if processing takes longer than the heartbeat interval, but it does need to finish before the session timeout expires or the consumer will be kicked out of the group (which typically is revealed by commit failures). If the problem is just that the consumer is handling too many messages at once, then Kafka 0.10 has an option to tune the number of messages returned from poll() (max.poll.records), which may be helpful. We also have a pause/resume API which allows you to call poll() without consuming any data. That's the best option at the moment for 0.9 consumers.
For what it's worth, we've considered several times adding a heartbeat() API, but the challenge is figuring out how to handle rebalancing. Underneath the covers, we use heartbeats to find out when the group is rebalancing, so a heartbeat() option would probably have to return a flag indicating whether a rebalance was needed. If the group has begun rebalancing, then you would need to call poll() before the expiration of the session timeout so that the consumer can join the rebalance. Alternatively, we could let heartbeat() complete the rebalance itself, but then you'd have to be prepared to abort processing from the rebalance callback. That's not really different from calling poll() after pausing partitions though. The main problem in any case is that once a rebalance begins, you have the duration of the session timeout to stop processing and join the rebalance. We're seeing this problem pop up pretty much everywhere that the consumer is used, so we're trying to think of some better options to handle it. Thanks, Jason On Thu, Apr 14, 2016 at 12:32 PM, Florian Hussonnois <fhussonn...@gmail.com> wrote: > Hi everyone, > > I have a use case where a message can take longer than ' > heartbeat.interval.ms' to be processed by my application. As I understand > the heartbeats of consumer are done while the poll method is invoked. > > I would like to instantiate a worker thread to process the messages but I > need to wait for the messages completion before polling again. > > Is there a way to force the consumer to make an heartbeat without polling > new messages ? > > Thanks, > > -- > Florian HUSSONNOIS >
