Hey Sumit, I thought you would be providing the actual steps to reproduce :) Nevertheless, can you get all the relevant logs: state change logs and controller logs at the very least and if possible server logs and send those over?
Joel On Tue, Feb 03, 2015 at 03:27:43PM -0800, Sumit Rangwala wrote: > In my setup kafka brokers are set for auto topic creation. In the scenario > below a node informs other nodes (currently 5 in total) about a number of > new (non-existent) topics, and all the nodes almost simultaneously open a > consumer for each of those topics. Sometime later another node informs all > other nodes of a new list of topics and each node, if they find that an > older topic exists in kafka, goes ahead and deletes the older topic. What I > have found is that many of the topics stay in the "marked for deletion" > state forever. > > > I get the list of topics using ZkUtils.getAllTopics(zkClient) and delete > topics using AdminUtils.deleteTopic(zkClient, topic). Since many nodes > might try to delete the same topic at the same time I do > see ZkNodeExistsException while deleting the topic, which I catch an > ignore. (e.g., org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = NodeExists for > /admin/delete_topics/LAX1-GRIFFIN-r13-1423001701601) > > # State of one deleted topic on kafka brokers: > Topic:LAX1-GRIFFIN-r13-1423001701601 PartitionCount:8 ReplicationFactor:1 > Configs: > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 0 Leader: -1 Replicas: > 49558 Isr: > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 1 Leader: -1 Replicas: > 49554 Isr: > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 2 Leader: -1 Replicas: > 49557 Isr: > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 3 Leader: -1 Replicas: > 49558 Isr: > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 4 Leader: -1 Replicas: > 49554 Isr: > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 5 Leader: -1 Replicas: > 49557 Isr: > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 6 Leader: -1 Replicas: > 49558 Isr: > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 7 Leader: -1 Replicas: > 49554 Isr: > > > # Controller log says > > [2015-02-03 22:59:03,399] INFO [delete-topics-thread-49554], Deletion for > replicas 49557,49554,49558 for partition > [LAX1-GRIFFIN-r13-1423001701601,0],[LAX1-GRIFFIN-r13-1423001701601,6],[LAX1-GRIFFIN-r13-1423001701601,5],[LAX1-GRIFFIN-r13-1423001701601,3],[LAX1-GRIFFIN-r13-1423001701601,7],[LAX1-GRIFFIN-r13-1423001701601,1],[LAX1-GRIFFIN-r13-1423001701601,4],[LAX1-GRIFFIN-r13-1423001701601,2] > of topic LAX1-GRIFFIN-r13-1423001701601 in progress > (kafka.controller.TopicDeletionManager$DeleteTopicsThread) > > current time: Tue Feb 3 23:20:58 UTC 2015 > > > Since I don't know the delete topic algorithm, I am not sure why sure these > topics are not garbage collected. I do have the complete setup running in > docker right now on my local box so please let me know if any more info is > required to troubleshoot this issue. > > > Furthermore, does ZkUtils.getAllTopics(zkClient) return "marked for > deletion" topic as well. If so, is there a easy way to get a list of active > topics (other than looking at all the topics in /admin/delete_topics/ and > taking a set difference with the topics returned > by ZkUtils.getAllTopics(zkClient) ) > > Sumit > (More setup info below) > > > Setup > -------- > Zookeeper: 3.4.6 > Kafka broker: 0.8.2-beta > Kafka clients: 0.8.2-beta > > # Kafka boker settings (all other settings are default 0.8.2-beta settings) > kafka.controlled.shutdown.enable: 'FALSE' > kafka.auto.create.topics.enable: 'TRUE' > kafka.num.partitions: 8 > kafka.default.replication.factor: 1 > kafka.rebalance.backoff.ms: 3000 > kafka.rebalance.max.retries: 10 > kafka.log.retention.minutes: 1200 > kafka.delete.topic.enable: 'TRUE'
