Hello Xin, The ZK write performance, especially latency, depends on the underlying hardware. AFAIK some organizations use SSD for their ZK clusters so that the latency is less than 1ms.
There are some more discussions on how many partitions one should choose in practice and what are ZK's impact on it: http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ Guozhang On Tue, Aug 16, 2016 at 10:18 AM, Xin Jin <xin...@cs.berkeley.edu> wrote: > Hi, > > I'm working on streaming systems in AMPLab at UC Berkeley. This article ( > https://cwiki.apache.org/confluence/display/KAFKA/Committing > +and+fetching+consumer+offsets+in+Kafka) > mentioned the ZooKeeper performance issue when consumers store offsets in > ZooKeeper. > > "In Kafka releases through 0.8.1.1, consumers commit their offsets to > ZooKeeper. ZooKeeper does not scale extremely well (especially for writes) > when there are a large number of offsets (i.e., consumer-count * > partition-count)." > > Can anyone tell me in production scenarios, how many consumers and > partitions do you have? How much write (offset update) traffic do you > generate that ZooKeeper cannot handle? > > Thank you very much! > Xin > -- -- Guozhang
