Tom, That is a good suggestion. Some of us started thinking about re-designing the consumer client a while ago and wrote up some ideas here - https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design. In addition to this, we have a working prototype of stage 1 of that re-design here https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Detailed+Consumer+Coordinator+Design
Besides this, work has started on scaling the offset storage for the consumer as part of this JIRA - https://issues.apache.org/jira/browse/KAFKA-657. It is true that the team is currently focussed on developing and stabilizing replication, but we welcome ideas and contribution to the consumer client re-design project as well. Thanks, Neha On Fri, Dec 21, 2012 at 5:44 PM, Tom Brown <tombrow...@gmail.com> wrote: > It seems that a common thread is that while ConsumerConnector works > well for the standard case, it just doesn't work for any case where > manual offset management (explicit checkpoints, rollbacks, etc) is > required. > > If any Kafka devs are looking for a way to improve it, I think > modifying it to be more modular regarding offset management would be > great! You could provide an interface for loading/committing offsets, > then provide a ZK implementation as the default. It would be backwards > compatible, but be useful in all of the use cases where explicit > offset management is required. > > (of course, I know I'm just an armchair kafka dev, so there may be > reasons why this won't work, or would be an extremely low priorty, > or...) > > --Tom > > On Fri, Dec 21, 2012 at 4:12 PM, Yonghui Zhao <zhaoyong...@gmail.com> > wrote: > > In our project we use senseidb to consume kafka data. Senseidb will > process the message immediately but won't flush to disk immeidately. So if > senseidb crash then all result not flushed will be lost， we want to rewind > kafka. The offset we want to rewind to is the flush checkpoint. > > In this case, we will lost some data > > > > Sent from my iPad > > > > 在 2012-12-22，1:37，Neha Narkhede <neha.narkh...@gmail.com> 写道： > > > >> > >> But if crash happens just after offset committed, then unprocessed > message in consumer will be skipped after reconnected. > >> > >> If the consumer crashes, you will get duplicates, not lose any data. > >> > >> Thanks, > >> Neha > >> >
