You can turn off automatic offset commit (auto.commit.enable=false) and use the commitOffsets() API. Note that this API will commit offsets for all partitions owned by the consumer.
Thanks, Neha On Nov 20, 2013 6:39 AM, "Imran Rashid" <im...@therashids.com> wrote: > Hi, > > I have an application which reads messages from a kafka queue, builds > up a batch of messages, and then performs some action on that batch. > So far I have just used the ConsumerGroup api. However, I realized > there is a potential problem -- my app may die sometime in the middle > of the batch, and then those messages could get completely lost. > Kafka will think we've already read those messages, and so on restart > it will skip to messages post batch. > > 1) Does the ConsumerGroup api have some way to allow you to control > when the the position of the consumer-group is updated? I couldn't > find anything relevant in ConsumerGroup, KafkaStream, or > ConsumerIterator, but perhaps I overlooked something. > > 2) Do I need to switch to SimpleConsumer? It looks like that does not > use zookeeper at all, I'd rather not have to manage all of the > zookeeper stuff myself. > > This seems like a basic use case, so I'm probably missing something ... > > thanks, > Imran > > > (I'm using 0.8.0-beta1) >
