I might not be the best one to answer this question. It will be better if someone has more operation experience can help out here.
Jiangjie (Becket) Qin On 4/21/15, 12:00 PM, "Achanta Vamsi Subhash" <achanta.va...@flipkart.com> wrote: >@Qin >Thanks for the inputs. Could you point us what we should monitor in >zookeeper if we have 20k partitions (this might go ~1 lakh in the coming 6 >months)? We are not running zookeeper on SSDs and we were facing >ZkConnectionLoss in HighLevelConsumer, sometimes for all the topics. What >are the critical things to monitor/optimise wrt. zookeeper. > >@Pushkar >Yes. We are using kafka for storing offsets instead of zookeeper. > >On Tue, Apr 21, 2015 at 11:55 PM, pushkar priyadarshi < >priyadarshi.push...@gmail.com> wrote: > >> In my knowledge if you are using 0.8.2.1 which is latest stable you can >> sync up your consumer offsets in kafka itself instead of Zk which >>further >> brings down write load on ZKs. >> >> Regards, >> Pushkar >> >> On Tue, Apr 21, 2015 at 1:13 PM, Jiangjie Qin >><j...@linkedin.com.invalid> >> wrote: >> >> > 20000 partitions should be OK. >> > >> > On 4/21/15, 12:33 AM, "Achanta Vamsi Subhash" < >> achanta.va...@flipkart.com> >> > wrote: >> > >> > >We are planning to have ~20000 partitions. Will it be a bottleneck? >> > > >> > >On Mon, Apr 20, 2015 at 10:48 PM, Jiangjie Qin >> <j...@linkedin.com.invalid >> > > >> > >wrote: >> > > >> > >> Producers usually do not query zookeeper at all. >> > >> Consumers usually query zookeeper at beginning or rebalance. It is >> > >> supposed to be in frequent if you don¹t have consumers come and go >>all >> > >>the >> > >> time. One exception is that if you are using zookeeper based >>consumer >> > >> offset commit, it will commit offset to zookeeper frequently. >> > >> In Kafka, the most heavily used mechanism for zookeeper is >>zookeeper >> > >> listener and they are not fired in a regular frequency. >> > >> >> > >> The limitation of Zookeeper usage for Kafka I am aware of is >>probably >> > >>the >> > >> size of each zNode. As long as you don¹t have so many partitions >>that >> > >> zNode cannot handle, it should be fine. >> > >> >> > >> Thanks. >> > >> >> > >> Jiangjie (Becket) Qin >> > >> >> > >> On 4/20/15, 5:58 AM, "Achanta Vamsi Subhash" >> > >><achanta.va...@flipkart.com> >> > >> wrote: >> > >> >> > >> >Hi, >> > >> > >> > >> >Could anyone help with this? >> > >> > >> > >> >Thanks. >> > >> > >> > >> >On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash < >> > >> >achanta.va...@flipkart.com> wrote: >> > >> > >> > >> >> Hi, >> > >> >> >> > >> >> How often does Kafka query zookeeper while producing and >>consuming? >> > >> >> >> > >> >> Ex: >> > >> >> If there is a single partition to which we produce and a >>HighLevel >> > >> >> consumer running on it, how many read/write queries to zookeeper >> > >>happen. >> > >> >> >> > >> >> Extending further, multiple topics with ~100 partitions each, >>how >> > >>many >> > >> >> zookeeper calls will be made (read/write). >> > >> >> >> > >> >> What is the max limit of no of partitions / kafka cluster that >> > >>zookeeper >> > >> >> can handle? >> > >> >> >> > >> >> -- >> > >> >> Regards >> > >> >> Vamsi Subhash >> > >> >> >> > >> > >> > >> > >> > >> > >> > >> >-- >> > >> >Regards >> > >> >Vamsi Subhash >> > >> >> > >> >> > > >> > > >> > >-- >> > >Regards >> > >Vamsi Subhash >> > >> > >> > > > >-- >Regards >Vamsi Subhash
