Hi, By design Kafka does ensure not to send same record to multiple consumers in same consumer group. Issue is because of rebalance while a processing is going on and records are not yet commited. In my view there are only 2 possible solutions to it 1) As mentioned in documentation, store offsets outside of kafka ( https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html). This is a complete solution but will definitely add extra developement and also extra processing to each message. Problem may still exist if at the time of a crash consumer was out of sync from external custom offset storage and offsets stored in kafka both. 2) As mentioned in fix for defect 919 ( https://issues.apache.org/jira/browse/KAFKA-919) set autocommit to true. This will make kafka commit fetched records before rebalancing. Only drawback is that some records may never be processed if consumer crashes while processing records which are already marked committed due to rebalance.
Regards, Vinay Sharma
