What's the controller value in the zk path (see https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper)? Any error in the controller/state-change log?
Thanks, Jun On Wed, Apr 16, 2014 at 10:07 AM, Seshadri, Balaji <balaji.sesha...@dish.com > wrote: > Hi, > > We got the following error spamming the logs when broker 1 is the > controller and we are shutting it down in controlled manner not kill -9. > > The leader being switched to broker 2 for all partitions but controller is > not being failed over to broker 2. > > [2014-04-16 10:48:47.976-0600] ERROR > [Controller-1-to-broker-1-send-thread], Controller 1's connection to broker > id:1,host:tm1-kafkabroker101,port:9092 was unsuccessful > (kafka.controller.RequestSendThread) > java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) > at sun.nio.ch.Net.connect(Net.java:465) > at sun.nio.ch.Net.connect(Net.java:457) > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) > at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) > at > kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173) > at > kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140) > at > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) > [2014-04-16 10:48:48.276-0600] ERROR > [Controller-1-to-broker-1-send-thread], Controller 1 epoch 38 failed to > send LeaderAndIsr request with correlation id 1766 to broker > id:1,host:tm1-kafkabroker101,port:9092. Reconnecting to broker. > (kafka.controller.RequestSendThread) > java.nio.channels.ClosedChannelException > at kafka.network.BlockingChannel.send(BlockingChannel.scala:89) > at > kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132) > at > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) > [2014-04-16 10:48:48.277-0600] ERROR > [Controller-1-to-broker-1-send-thread], Controller 1's connection to broker > id:1,host:tm1-kafkabroker101,port:9092 was unsuccessful > (kafka.controller.RequestSendThread) > java.net.ConnectException: Connection refused > at sun.nio.ch.Net.connect0(Native Method) > at sun.nio.ch.Net.connect(Net.java:465) > at sun.nio.ch.Net.connect(Net.java:457) > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) > at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) > at > kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173) > at > kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140) > at > kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) > > > It stopped our upgrade because of this issue. > > Thanks, > > Balaji > > >
