Hi,

I have an application which reads messages from a kafka queue, builds
up a batch of messages, and then performs some action on that batch.
So far I have just used the ConsumerGroup api.  However, I realized
there is a potential problem -- my app may die sometime in the middle
of the batch, and then those messages could get completely lost.
Kafka will think we've already read those messages, and so on restart
it will skip to messages post batch.

1) Does the ConsumerGroup api have some way to allow you to control
when the the position of the consumer-group is updated?  I couldn't
find anything relevant in ConsumerGroup, KafkaStream, or
ConsumerIterator, but perhaps I overlooked something.

2) Do I need to switch to SimpleConsumer?  It looks like that does not
use zookeeper at all, I'd rather not have to manage all of the
zookeeper stuff myself.

This seems like a basic use case, so I'm probably missing something ...

thanks,
Imran


(I'm using 0.8.0-beta1)

Reply via email to