Awesome! Thanks for the clarification. I would like to offer my strong vote that this get tackled before a beta, to get it firmly into 0.8. Stabilize everything else to the existing use, but make offset updates batched.
thanks, rob ________________________________________ From: Neha Narkhede [neha.narkh...@gmail.com] Sent: Friday, May 17, 2013 7:17 AM To: users@kafka.apache.org Subject: RE: are commitOffsets botched to zookeeper? Sorry I wasn't clear. Zookeeper 3.4.x has this feature. As soon as 08 is stable and released it will be worth looking into when we can use zookeeper 3.4.x. Thanks, Neha On May 16, 2013 10:32 PM, "Rob Withers" <reefed...@gmail.com> wrote: > Can a request be made to zookeeper for this feature? > > Thanks, > rob > > > -----Original Message----- > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > Sent: Thursday, May 16, 2013 9:53 PM > > To: users@kafka.apache.org > > Subject: Re: are commitOffsets botched to zookeeper? > > > > Currently Kafka depends on zookeeper 3.3.4 that doesn't have a batch > write > > api. So if you commit after every message at a high rate, it will be slow > and > > inefficient. Besides it will cause zookeeper performance to degrade. > > > > Thanks, > > Neha > > On May 16, 2013 6:54 PM, "Rob Withers" <reefed...@gmail.com> wrote: > > > > > We are calling commitOffsets after every message consumption. It > > > looks to be ~60% slower, with 29 partitions. If a single KafkaStream > > > thread is from a connector, and there are 29 partitions, then > > > commitOffsets sends 29 offset updates, correct? Are these offset > > > updates batched in one send to zookeeper? > > > > > > thanks, > > > rob > >
