Re: Rebalancing during the long-running tasks

Great proposal, indeed. If I understand right, Jason suggests committing messages one-by-one instead of ingesting the whole batch. I have nothing against processing the big bunch of tasks from poll() and committing after every single processed task, but in my case the very single task is too hea

Re: Rebalancing during the long-running tasks

Hi, I had the same issue and managed to work around it by simulating a heartbeat to kafka. It works really well, i.e., we have had zero issues since it was implemented I have somthing like this: void process() { records = consumer.poll(timeout) dispatcher.dispatch(records) while(!dispa

Re: Rebalancing during the long-running tasks

I think you’ll find some useful context in this KIP Jason wrote. It’s pretty good. https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records > On 16 Feb 2016, at 07:15, Насыров Р

Rebalancing during the long-running tasks

Hello! I'm trying to use kafka for long-running tasks processing. The tasks can be very short (less than a second) or very long (about 10 minutes). I've got one consumer group for the single queue, and one or more consumers. Sometimes consumers manage to commit their offsets before rebalancing,