Controller is not being failed over 0.8.1

[Seshadri, Balaji]

Hi,

We got the following error spamming the logs when broker 1 is the controller 
and we are shutting it down in controlled manner not kill -9.

The leader being switched to broker 2 for all partitions but controller is not 
being failed over to broker 2.

[2014-04-16 10:48:47.976-0600] ERROR [Controller-1-to-broker-1-send-thread], 
Controller 1's connection to broker id:1,host:tm1-kafkabroker101,port:9092 was 
unsuccessful (kafka.controller.RequestSendThread)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect0(Native Method)
        at sun.nio.ch.Net.connect(Net.java:465)
        at sun.nio.ch.Net.connect(Net.java:457)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at 
kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
        at 
kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
        at 
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-04-16 10:48:48.276-0600] ERROR [Controller-1-to-broker-1-send-thread], 
Controller 1 epoch 38 failed to send LeaderAndIsr request with correlation id 
1766 to broker id:1,host:tm1-kafkabroker101,port:9092. Reconnecting to broker. 
(kafka.controller.RequestSendThread)
java.nio.channels.ClosedChannelException
        at kafka.network.BlockingChannel.send(BlockingChannel.scala:89)
        at 
kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132)
        at 
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
[2014-04-16 10:48:48.277-0600] ERROR [Controller-1-to-broker-1-send-thread], 
Controller 1's connection to broker id:1,host:tm1-kafkabroker101,port:9092 was 
unsuccessful (kafka.controller.RequestSendThread)
java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect0(Native Method)
        at sun.nio.ch.Net.connect(Net.java:465)
        at sun.nio.ch.Net.connect(Net.java:457)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
        at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
        at 
kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:173)
        at 
kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:140)
        at 
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)


It stopped our upgrade because of this issue.

Thanks,

Balaji