EDIT: At the end of the controller.log logs the following line should be added:
ERROR [Controller 100]: Error while electing or becoming controller on broker 100 (kafka.controller.KafkaController) java.util.NoSuchElementException: key not found: 0 On Sat, Jan 5, 2019 at 4:21 PM Soheil Pourbafrani <soheil.i...@gmail.com> wrote: > Hi, using Kafka 11.0.3 I set up a 2 node Kafka cluster on CentOS 7 with > broker id 100 and 101. There is no firewall enabled. Both nodes can ping > each other and establish the SSH connection successfully. > When I start the cluster both broker successfully register their id into > the zookeeper cluster (I checked the path /brokers/ids) and the command to > list the topics can be run successfully. > > But monitoring brokers logs I see it continuously logs in the file > server.log on the broker with id 100 : > INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral) > WARN Connection to node 101 could not be established. Broker may not be > available. (org.apache.kafka.clients.Net$ > INFO Creating /controller (is it secure? false) > (kafka.utils.ZKCheckedEphemeral) > > Also in the controller.log file on the broker id 100, it logs continuously: > INFO [Controller 100]: Broker 100 resigned as the controller > (kafka.controller.KafkaController) > DEBUG [Controller 100]: Controller resigning, broker id 100 > (kafka.controller.KafkaController) > DEBUG [Controller 100]: De-registering IsrChangeNotificationListener > (kafka.controller.KafkaController) > INFO [Partition state machine on Controller 100]: Stopped partition state > machine (kafka.controller.PartitionStateMachine) > INFO [Replica state machine on controller 100]: Stopped replica state > machine (kafka.controller.ReplicaStateMachine) > INFO [Controller 100]: Broker 100 resigned as the controller > (kafka.controller.KafkaController) > INFO [Controller 100]: 100 successfully elected as the controller > (kafka.controller.KafkaController) > INFO [Controller 100]: Broker 100 starting become controller state > transition (kafka.controller.KafkaController) > INFO [Controller 100]: Initialized controller epoch to 96100 and zk > version 96099 (kafka.controller.KafkaController) > INFO [Controller 100]: Controller 100 incremented epoch to 96101 > (kafka.controller.KafkaController) > DEBUG [Controller 100]: Registering IsrChangeNotificationListener > (kafka.controller.KafkaController) > DEBUG [Channel manager on controller 100]: Controller 100 trying to > connect to broker 100 > DEBUG [Channel manager on controller 100]: Controller 100 trying to > connect to broker 101 > > I also check the network packets using tcpdump and I observed > Zookeeper nodes could get packets from both brokers successfully, but there > isn't any packet has been sent from each broker to the other. > > I checked the established connections on both brokers using netstat and > I observed there are much TIME_WAIT established connections between two > brokers. > > What could be the reason? > > >
