Hey Guven, This problem is what KIP-41 was created for: https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records .
The patch for this was committed yesterday and will be included in 0.10. If you need something in the shorter term, you could probably use the client from trunk (no changes to the server are needed). If this is still not sufficient, I recommend looking into the pause() API, which can facilitate asynchronous message processing in another thread. -Jason On Thu, Feb 25, 2016 at 8:53 AM, Guven Demir <[email protected]> wrote: > hi all, > > i'm having trouble processing a topic which includes paths to images which > need to be downloaded and saved to disk (each takes ~3-5 seconds) and > several are received on each poll > > within this scenario, i'm receiving the following error: > > org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot > be completed due to group rebalance > > which i assume is due to heartbeat failure and broker re-assigning the > consumer's partition to another consumer > > are there any recommendations for processing long to process messages? > > thanks in advance, > guven > > >
