Re: ZooKeeper performance issue for storing offset

2016-08-19 Thread Guozhang Wang
Hello Xin, The ZK write performance, especially latency, depends on the underlying hardware. AFAIK some organizations use SSD for their ZK clusters so that the latency is less than 1ms. There are some more discussions on how many partitions one should choose in practice and what are ZK's impact o

ZooKeeper performance issue for storing offset

2016-08-16 Thread Xin Jin
Hi, I'm working on streaming systems in AMPLab at UC Berkeley. This article ( https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka) mentioned the ZooKeeper performance issue when consumers store offsets in ZooKeeper. "In Kafka releases through 0.8.1.