Hi All, We are having 8 broker kafka cluster configured in our setup and created a topic with 8 partitions & 3 replicas. While trying to consume data from one broker(id:6) we are getting below errors and increase in lag for active partition on that host. We even tried restarting, deleting logs and reinstalling broker but still no luck. This is happening only for one broker. If we shut down that broker service everything works well. Kafka Server Logs: [2017-07-31 19:32:39,212] INFO [Group Metadata Manager on Broker 6]: Finished loading offsets from __consumer_offsets-5 in 5 milliseconds. (kafka.coordinator.GroupMe tadataManager) [2017-07-31 19:32:39,217] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-13 (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,218] INFO [Group Metadata Manager on Broker 6]: Loading offsets and group metadata from __consumer_offsets-13 (kafka.coordinator.GroupMetadataMa nager) [2017-07-31 19:32:39,220] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-37 (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,230] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-45 (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,232] INFO [Group Metadata Manager on Broker 6]: Finished loading offsets from __consumer_offsets-13 in 13 milliseconds. (kafka.coordinator.Group MetadataManager) [2017-07-31 19:32:39,232] INFO [Group Metadata Manager on Broker 6]: Loading offsets and group metadata from __consumer_offsets-37 (kafka.coordinator.GroupMetadataMa nager) [2017-07-31 19:32:39,248] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions bseEquityAggFeedProd-4 (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,249] INFO Truncating log bseEquityAggFeedProd-4 to offset 13082598. (kafka.log.Log) [2017-07-31 19:32:39,276] INFO [ReplicaFetcherManager on broker 6] Added fetcher for partitions List([bseEquityAggFeedProd-4, initOffset 13082598 to broker BrokerEnd Point(2,172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,277] INFO [ReplicaFetcherThread-0-2], Starting (kafka.server.ReplicaFetcherThread) [2017-07-31 19:32:39,294] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-17 (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,294] INFO Truncating log __consumer_offsets-17 to offset 0. (kafka.log.Log) [2017-07-31 19:32:39,300] INFO [ReplicaFetcherManager on broker 6] Added fetcher for partitions List([__consumer_offsets-17, initOffset 0 to broker BrokerEndPoint(2, 172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,304] ERROR [ReplicaFetcherThread-0-5], Error for partition [__consumer_offsets,9] to broker 5:org.apache.kafka.common.errors.NotLeaderForPartiti onException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-07-31 19:32:39,308] ERROR [ReplicaFetcherThread-0-2], Error for partition [bseEquityAggFeedProd,4] to broker 2:org.apache.kafka.common.errors.NotLeaderForParti tionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-07-31 19:32:39,313] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,17] to broker 2:org.apache.kafka.common.errors.NotLeaderForPartit ionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-07-31 19:32:39,318] INFO [ReplicaFetcherManager on broker 6] Removed fetcher for partitions __consumer_offsets-9 (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,319] INFO Truncating log __consumer_offsets-9 to offset 392. (kafka.log.Log) [2017-07-31 19:32:39,324] INFO [ReplicaFetcherManager on broker 6] Added fetcher for partitions List([__consumer_offsets-9, initOffset 392 to broker BrokerEndPoint(2 ,172.29.51.52,9092)] ) (kafka.server.ReplicaFetcherManager) [2017-07-31 19:32:39,339] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,9] to broker 2:org.apache.kafka.common.errors.NotLeaderForPartiti onException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread) [2017-07-31 19:32:39,986] INFO [GroupCoordinator 6]: Loading group metadata for bseEquityLegacyBod with generation 4 (kafka.coordinator.GroupCoordinator) [2017-07-31 19:32:39,991] INFO [Group Metadata Manager on Broker 6]: Finished loading offsets from __consumer_offsets-37 in 759 milliseconds. (kafka.coordinator.Grou pMetadataManager)
Consumer Logs: 17/07/31/19:26:20.005 2-thread-3 INFO nals.AbstractCoordinator Discovered coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) for group bseEquityLegacyBod. 17/07/31/19:26:20.005 2-thread-3 INFO nals.AbstractCoordinator Marking the coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) dead for group bseEquityLegacyBod 17/07/31/19:26:20.106 2-thread-3 INFO nals.AbstractCoordinator Discovered coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) for group bseEquityLegacyBod. 17/07/31/19:26:20.106 2-thread-3 INFO nals.AbstractCoordinator Marking the coordinator 172.XX.51.56:9092 (id: 2147483641 rack: null) dead for group bseEquityLegacyBod Thanks in advance. Regards, Mahesh Patade ________________________________ DISCLAIMER AND PRIVILEGE NOTICE : This e-mail and any files transmitted with it contain confidential, copyright, proprietary and legally privileged information. It should not be used by anyone who is not the original intended recipient. Any use, distribution, copying or disclosure by any other person is strictly prohibited. If you receive this transmission in error, please notify the sender by reply email and then destroy the message. Opinions, conclusions and other information in this message that do not relate to official business of Network18 Media & Investments Ltd., its subsidiaries, holding companies i.e. Network18 group shall be understood to be neither given nor endorsed by Network18 group. Internet communications cannot be guaranteed to be timely, Secure, error or virus-free. The sender does not accept liability for any errors or omissions.
