Hi * My kafka setup
**OS: Windows Machine*6 broker nodes , 4 on one Machine and 2 on other Machine* **ZK instance on (4 broker nodes Machine) and another ZK on (2 broker nodes machine)* ** 2 Topics with partition size = 50 and replication factor = 3* I am producing on an average of around 500 messages / sec with each message size close to 98 bytes... More or less the message rate stays constant throughout, but after running the setup for close to 2 weeks , my Kafka cluster broke and this happened twice in a month. Not able to understand what's the issue, Kafka gurus please do share your inputs... the controlle.log file at the time of Kafka broken looks like *[2017-04-26 12:03:34,998] INFO [Controller 0]: Broker failure callback for 0,1,3,5,6 (kafka.controller.KafkaController)[2017-04-26 12:03:34,998] INFO [Controller 0]: Removed ArrayBuffer() from list of shutting down brokers. (kafka.controller.KafkaController)[2017-04-26 12:03:34,998] INFO [Partition state machine on Controller 0]: Invoking state change to OfflinePartition for partitions [__consumer_offsets,19],[mytopic,11],[__consumer_offsets,30],[mytopicOLD,18],[mytopic,13],[__consumer_offsets,47],[mytopicOLD,26],[__consumer_offsets,29],[mytopicOLD,0],[__consumer_offsets,41],[mytopic,44],[mytopicOLD,38],[mytopicOLD,2],[__consumer_offsets,17],[__consumer_offsets,10],[mytopic,20],[mytopic,23],[mytopic,30],[__consumer_offsets,14],[__consumer_offsets,40],[mytopic,31],[mytopicOLD,43],[mytopicOLD,19],[mytopicOLD,35],[__consumer_offsets,18],[mytopic,43],[__consumer_offsets,26],[__consumer_offsets,0],[mytopic,32],[__consumer_offsets,24],[mytopicOLD,3],[mytopic,2],[mytopic,3],[mytopicOLD,45],[mytopic,35],[__consumer_offsets,20],[mytopic,1],[mytopicOLD,33],[__consumer_offsets,5],[mytopicOLD,47],[__consumer_offsets,22],[mytopicOLD,8],[mytopic,33],[mytopic,36],[mytopicOLD,11],[mytopic,47],[mytopicOLD,20],[mytopic,48],[__consumer_offsets,12],[mytopicOLD,32],[__consumer_offsets,8],[mytopicOLD,39],[mytopicOLD,27],[mytopicOLD,49],[mytopicOLD,42],[mytopic,21],[mytopicOLD,31],[mytopic,29],[__consumer_offsets,23],[mytopicOLD,21],[__consumer_offsets,48],[__consumer_offsets,11],[mytopic,18],[__consumer_offsets,13],[mytopic,45],[mytopic,5],[mytopicOLD,25],[mytopic,6],[mytopicOLD,23],[mytopicOLD,37],[__consumer_offsets,6],[__consumer_offsets,49],[mytopicOLD,13],[__consumer_offsets,28],[__consumer_offsets,4],[__consumer_offsets,37],[mytopic,12],[mytopicOLD,30],[__consumer_offsets,31],[__consumer_offsets,44],[mytopicOLD,15],[mytopicOLD,29],[mytopic,37],[mytopic,38],[__consumer_offsets,42],[mytopic,27],[mytopic,26],[mytopic,15],[__consumer_offsets,34],[mytopic,42],[__consumer_offsets,46],[mytopic,14],[mytopicOLD,12],[mytopicOLD,1],[mytopic,7],[__consumer_offsets,25],[mytopicOLD,24],[mytopicOLD,44],[mytopicOLD,14],[__consumer_offsets,32],[mytopic,0],[__consumer_offsets,43],[mytopic,39],[mytopicOLD,5],[mytopic,9],[mytopic,24],[__consumer_offsets,36],[mytopic,25],[mytopicOLD,36],[mytopic,19],[__consumer_offsets,35],[__consumer_offsets,7],[mytopic,8],[__consumer_offsets,38],[mytopicOLD,48],[mytopicOLD,9],[__consumer_offsets,1],[mytopicOLD,6],[mytopic,41],[mytopicOLD,41],[mytopicOLD,7],[mytopic,17],[mytopicOLD,17],[mytopic,49],[__consumer_offsets,16],[__consumer_offsets,2] (kafka.controller.PartitionStateMachine)[2017-04-26 12:03:35,045] INFO [SessionExpirationListener on 1], ZK expired; shut down all controller components and try to re-elect (kafka.controller.KafkaController$SessionExpirationListener)[2017-04-26 12:03:35,045] DEBUG [Controller 1]: Controller resigning, broker id 1 (kafka.controller.KafkaController)[2017-04-26 12:03:35,045] DEBUG [Controller 1]: De-registering IsrChangeNotificationListener (kafka.controller.KafkaController)[2017-04-26 12:03:35,060] INFO [Partition state machine on Controller 1]: Stopped partition state machine (kafka.controller.PartitionStateMachine)[2017-04-26 12:03:35,060] INFO [Replica state machine on controller 1]: Stopped replica state machine (kafka.controller.ReplicaStateMachine)[2017-04-26 12:03:35,060] INFO [Controller 1]: Broker 1 resigned as the controller (kafka.controller.KafkaController)[2017-04-26 12:03:36,013] DEBUG [OfflinePartitionLeaderSelector]: No broker in ISR is alive for [__consumer_offsets,19]. Pick the leader from the alive assigned replicas: (kafka.controller.OfflinePartitionLeaderSelector)[2017-04-26 12:03:36,029] DEBUG [OfflinePartitionLeaderSelector]: No broker in ISR is alive for [mytopic,11]. Pick the leader from the alive assigned replicas: (kafka.controller.OfflinePartitionLeaderSelector)[2017-04-26 12:03:36,029] DEBUG [OfflinePartitionLeaderSelector]: No broker in ISR is alive for [__consumer_offsets,30]. Pick the leader from the alive assigned replicas: (kafka.controller.OfflinePartitionLeaderSelector)[2017-04-26 12:03:37,811] DEBUG [OfflinePartitionLeaderSelector]: Some broker in ISR is alive for [mytopicOLD,18]. Select 2 from ISR 2 to be the leader. (kafka.controller.OfflinePartitionLeaderSelector)* Typical broker config attached.. Please do share some valid inputs... Abhi !wq *-- * If you can't succeed, call it version 1.0
broker.id=0 port:9093 num.network.threads=3 num.io.threads=8 server socket.send.buffer.bytes=102400 server socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 num.recovery.threads.per.data.dir=1 num.partitions=1 log.retention.minutes=360 auto.create.topics.enable=false log.segment.bytes=52428800 log.cleaner.enable=true log.cleanup.policy=delete log.cleaner.min.cleanable.ratio=0.5 log.cleaner.backoff.ms=15000 log.segment.delete.delay.ms=6000 auto.create.topics.enable=false log.retention.check.interval.ms=300000
