Ah, yes those consumers znodes in Zookeeper are related to consumer group metadata. You can actually find a full write up of what each znode is used for here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
Cheers, On Wed, May 11, 2016 at 9:22 AM, Spico Florin <spicoflo...@gmail.com> wrote: > Hi, Dustin! > Thank you for your answer. I have observed that in zookeeper there was a > folder named consumers, that kept data about the topic name its partitions > and the offsets. > Were both the consumers/producers using this folder to keep track of the > offsets? What was the purpose of this folder? > I look forward for your answers. > Regards, > Florin > > On Wed, May 11, 2016 at 4:12 PM, Dustin Cote <dus...@confluent.io> wrote: > > > Hi Florin, > > > > The new consumer is intended to replace both the high level and simple > > consumers. > > > > http://kafka.apache.org/documentation.html#consumerapi > > > > The old simple consumer API didn't rely on zookeeper to store offsets, > but > > rather the client was responsible for managing their own offsets as there > > was no consumer group associated to a simple consumer. However, your > > simple consumer did talk to zookeeper to find out about partition leaders > > and there is some handling required there. > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example > > > > I think Gwen's answer is complete in that respect. Hope that helps. > > > > On Wed, May 11, 2016 at 3:23 AM, Spico Florin <spicoflo...@gmail.com> > > wrote: > > > > > Hi! > > > In Gwen's answer there is nothing related with the simple API consumer > > and > > > the relation with ZK. But still the simple consumer API uses ZK to > store > > > offsets, isn't it? > > > I look forward for your answer. > > > Regards, > > > Florin > > > > > > On Wed, May 11, 2016 at 2:11 AM, R Krishna <krishna...@gmail.com> > wrote: > > > > > > > And where is the documentation for this topic: "__consumers_offsets" > > > > > > > > On Tue, May 10, 2016 at 1:16 AM, Spico Florin <spicoflo...@gmail.com > > > > > > wrote: > > > > > > > > > Hi! > > > > > Yes both are possible. The new versions 0.9 and above store the > > > offsets > > > > in > > > > > a special Kafka topic named __consumers_offsets. > > > > > Regards, > > > > > florin > > > > > > > > > > On Tue, May 10, 2016 at 8:33 AM, Gerard Klijs < > > gerard.kl...@dizzit.com > > > > > > > > > wrote: > > > > > > > > > > > Both are possible, but the 'new' consumer stores the offset in an > > > > > __offset > > > > > > topic. > > > > > > > > > > > > On Tue, May 10, 2016 at 7:07 AM Snehalata Nagaje < > > > > > > snehalata.nag...@harbingergroup.com> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > As per kafka 9 version, where does kafka store committed > offset? > > > > > > > > > > > > > > is it in zookeeper or kafka broker? > > > > > > > > > > > > > > Also there is option to use offset storage outside kafka, does > it > > > > mean > > > > > , > > > > > > > kafka will not depend on zookeepr for offset. > > > > > > > > > > > > > > Thanks, > > > > > > > snehalata > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Radha Krishna, Proddaturi > > > > 253-234-5657 > > > > > > > > > > > > > > > -- > > Dustin Cote > > confluent.io > > > -- Dustin Cote confluent.io
