Re: Kafka cluster instablility

2018-02-14 Thread Ted Yu
For #2 and #3, you would get better stability if zookeeper and Kafka get dedicated machines. Have you profiled the performance of the nodes where multiple processes ran (zookeeper / Kafka / Druid) ? How was disk and network IO like ? Cheers On Wed, Feb 14, 2018 at 9:38 AM, Avinash Herle wrote:

Kafka cluster instablility

2018-02-14 Thread Avinash Herle
Hi, I'm using Kafka version 0.11.0.2. In my cluster, I've 4 nodes running Kafka of which 3 nodes also running Zookeeper. I've a few producer processes that publish to Kafka and multiple consumer processes, a streaming engine (Spark) that ingests from Kafka and also publishes data to Kafka, and a d