Yes, the new consumer api will solve your probably better. Before that's ready, another option is to use the commitOffset() api in the high level consumer. It doesn't take any offset though. So, to prevent message loss during consumer failure, you will need to make sure all iterated messages are fully processed before calling commitOffset().
Thanks, Jun On Wed, Nov 12, 2014 at 11:35 AM, Falabella, Anthony < anthony.falabe...@citi.com> wrote: > Hi Jun, > > Thanks for taking a look at my issue and also for updating the future > release plan Wiki page. > > My use case is to use Kafka as if it were a JMS provider (messaging use > case). > I'm currently using Kafka 0.8.1.1 with Java and specifically the Spring > Integration Kafka Inbound Channel Adapter. Internally that adapter uses > the HighLevelConsumer which shields the caller from the internals of > offsets. Let's take the case where a consumer-group reads a number of > messages and then is abruptly terminated before properly processing those > messages. In that scenario upon restart ideally we'd begin reading at the > offset we were at prior to abruptly terminating. If we have > "auto.commit.enable=true" upon restart those messages will be considered > already read and will be skipped. Setting "auto.commit.enable=false" would > help in this case but now we'd have to manually call on the offset manager, > requiring the use of the SimpleConsumer. > > In my use-case, it's acceptable for some manual intervention to say > "reprocess messaging X thru Y", but to do so would require us to know the > exact offset we had started at prior to the chunk that was read in when the > JVM abnormally terminated. Perhaps I could look at the underlying > ExportZkOffsets and ImportZkOffsets Java classes mentioned in this link, > but at the very least I'd need to log the timestamp just prior to my read > to be used in that query per: > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Idon'twantmyconsumer'soffsetstobecommittedautomatically.CanImanuallymanagemyconsumer'soffsets > ? > > It sounds like the ConsumerAPI rewrite mentioned in these links might be > helpful in my situation (potentially targeted for Apr 2015): > > https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design > > In the meantime if you have any suggestions for things I might be able to > use to work-around my concern I'd be appreciative. Again I'm on 0.8.1.1 > but would be willing to look at 0.8.2 if it offered anything to help with > my use-case. > > Thanks > Tony >
