Sam Meder created KAFKA-1447:
--------------------------------
Summary: Controlled shutdown deadlock when trying to send state
updates
Key: KAFKA-1447
URL: https://issues.apache.org/jira/browse/KAFKA-1447
Project: Kafka
Issue Type: Bug
Components: controller
Affects Versions: 0.8.0
Reporter: Sam Meder
Assignee: Neha Narkhede
We're seeing controlled shutdown indefinitely stuck on trying to send out state
change messages to the other brokers:
[2014-05-03 04:01:30,580] INFO [Socket Server on Broker 4], Shutdown completed
(kafka.network.SocketServer)
[2014-05-03 04:01:30,581] INFO [Kafka Request Handler on Broker 4], shutting
down (kafka.server.KafkaRequestHandlerPool)
and stuck on:
"kafka-request-handler-12" daemon prio=10 tid=0x00007f1f04a66800 nid=0x6e79
waiting on condition [0x00007f1ad5767000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
parking to wait for <0x000000078e91dc20> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)
at
kafka.controller.ControllerChannelManager.sendRequest(ControllerChannelManager.scala:57)
locked <0x000000078e91dc38> (a java.lang.Object)
at kafka.controller.KafkaController.sendRequest(KafkaController.scala:655)
at
kafka.controller.ControllerBrokerRequestBatch$$anonfun$sendRequestsToBrokers$2.apply(ControllerChannelManager.scala:298)
at
kafkler.ControllerBrokerRequestBatch$$anonfun$sendRequestsToBrokers$2.apply(ControllerChannelManager.scala:290)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
at scala.collection.Iterator$class.foreach(Iterator.scala:772)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
at
kafka.controller.ControllerBrokerRequestBatch.sendRequestsToBrokers(ControllerChannelManager.scala:290)
at
kafka.controller.ReplicaStateMachine.handleStateChanges(ReplicaStateMachine.scala:97)
at
kafka.controller.KafkaController$$anonfun$shutdownBroker$3$$anonfun$apply$1$$anonfun$apply$mcV$sp$3.apply(KafkaController.scala:269)
at
kafka.controller.KafkaController$$anonfun$shutdownBroker$3$$anonfun$apply$1$$anonfun$apply$mcV$sp$3.apply(KafkaController.scala:253)
at scala.Option.foreach(Option.scala:197)
at
kafka.controller.KafkaController$$anonfun$shutdownBroker$3$$anonfun$apply$1.apply$mcV$sp(KafkaController.scala:253)
at
kafka.controller.KafkaController$$anonfun$shutdownBroker$3$$anonfun$apply$1.apply(KafkaController.scala:253)
at
kafka.controller.KafkaController$$anonfun$shutdownBroker$3$$anonfun$apply$1.apply(KafkaController.scala:253)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at
kafka.controller.KafkaController$$anonfun$shutdownBroker$3.apply(KafkaController.scala:252)
at
kafka.controller.KafkaController$$anonfun$shutdownBroker$3.apply(KafkaController.scala:249)
at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:130)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:275)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:275)
at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:275)
at kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:249)
locked <0x000000078b495af0> (a java.lang.Object)
at kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:264)
at kafka.server.KafkaApis.handle(KafkaApis.scala:192)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:722)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
