Re: Zookeeper load burst during replication

2014-12-03 Thread Yury Ruchin
Hi Guozhang, No consumer clients were consuming when the broker was replicating, at least none that would commit offsets to ZK. As for the controller log, that's a good idea, I will try it out. Thanks! 2014-12-02 22:28 GMT+03:00 Guozhang Wang : > Kafka brokers uses ZK for metadata storage, and K

Re: Zookeeper load burst during replication

2014-12-02 Thread Guozhang Wang
Kafka brokers uses ZK for metadata storage, and Kafka consumer clients uses ZK for offset and member management. For metadata storage, when there is replica state changes (for example like the new replica added after a broker restart in your case) the controller will try to write to ZK recording s

Zookeeper load burst during replication

2014-12-02 Thread Yury Ruchin
Hello, In a multi-broker Kafka 0.8.1.1 setup, I had one broker crashed. I restarted it after some noticeable time, so it started catching up the leader very intensively. During the replication, I see that the disk load on the ZK leader bursts abnormally, resulting in ZK performance degradation. Wh