Re: Kafka Errors and Hung Brokers

2016-12-14 Thread Shailesh Hemdev
Hi Apurva, The first error vanished after I restarted all the brokers. I haven't seen these recursive errors and my thought is since we restarted zookeeper nodes we might have put all the brokers in some sort of a iffy state The broker occasionally being hung has plagued us quite a bit. Our Kafka

Re: Kafka Errors and Hung Brokers

2016-12-14 Thread Apurva Mehta
Regarding 1), you can see a NotLeaderForPartition exception if the leader for the partition has moved to another host but the client metadata has not updated itself yet. The messages should disappear once the metadata is updated on all clients. Leaders may move if brokers are bounced, or if they h

Kafka Errors and Hung Brokers

2016-12-13 Thread Shailesh Hemdev
We are using a 3 node Kafka cluster and are encountering some weird issues. 1) On Each node, when we tail the server.log file under /var/log/kafka we see continuous errors like these pic-partition. (kafka.server.ReplicaFetcherThread) [2016-12-14 02:39:30,747] ERROR [ReplicaFetcherThread-0-441

Kafka Errors and Hung Brokers

2016-12-13 Thread Shailesh Hemdev
We are using a 3 node Kafka cluster and are encountering some weird issues. 1) On Each node, when we tail the server.log file under /var/log/kafka we see continuous errors like these pic-partition. (kafka.server.ReplicaFetcherThread) [2016-12-14 02:39:30,747] ERROR [ReplicaFetcherThread-0-441