In my setup kafka brokers are set for auto topic creation. In the scenario below a node informs other nodes (currently 5 in total) about a number of new (non-existent) topics, and all the nodes almost simultaneously open a consumer for each of those topics. Sometime later another node informs all other nodes of a new list of topics and each node, if they find that an older topic exists in kafka, goes ahead and deletes the older topic. What I have found is that many of the topics stay in the "marked for deletion" state forever.
I get the list of topics using ZkUtils.getAllTopics(zkClient) and delete topics using AdminUtils.deleteTopic(zkClient, topic). Since many nodes might try to delete the same topic at the same time I do see ZkNodeExistsException while deleting the topic, which I catch an ignore. (e.g., org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /admin/delete_topics/LAX1-GRIFFIN-r13-1423001701601) # State of one deleted topic on kafka brokers: Topic:LAX1-GRIFFIN-r13-1423001701601 PartitionCount:8 ReplicationFactor:1 Configs: Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 0 Leader: -1 Replicas: 49558 Isr: Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 1 Leader: -1 Replicas: 49554 Isr: Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 2 Leader: -1 Replicas: 49557 Isr: Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 3 Leader: -1 Replicas: 49558 Isr: Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 4 Leader: -1 Replicas: 49554 Isr: Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 5 Leader: -1 Replicas: 49557 Isr: Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 6 Leader: -1 Replicas: 49558 Isr: Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 7 Leader: -1 Replicas: 49554 Isr: # Controller log says [2015-02-03 22:59:03,399] INFO [delete-topics-thread-49554], Deletion for replicas 49557,49554,49558 for partition [LAX1-GRIFFIN-r13-1423001701601,0],[LAX1-GRIFFIN-r13-1423001701601,6],[LAX1-GRIFFIN-r13-1423001701601,5],[LAX1-GRIFFIN-r13-1423001701601,3],[LAX1-GRIFFIN-r13-1423001701601,7],[LAX1-GRIFFIN-r13-1423001701601,1],[LAX1-GRIFFIN-r13-1423001701601,4],[LAX1-GRIFFIN-r13-1423001701601,2] of topic LAX1-GRIFFIN-r13-1423001701601 in progress (kafka.controller.TopicDeletionManager$DeleteTopicsThread) current time: Tue Feb 3 23:20:58 UTC 2015 Since I don't know the delete topic algorithm, I am not sure why sure these topics are not garbage collected. I do have the complete setup running in docker right now on my local box so please let me know if any more info is required to troubleshoot this issue. Furthermore, does ZkUtils.getAllTopics(zkClient) return "marked for deletion" topic as well. If so, is there a easy way to get a list of active topics (other than looking at all the topics in /admin/delete_topics/ and taking a set difference with the topics returned by ZkUtils.getAllTopics(zkClient) ) Sumit (More setup info below) Setup -------- Zookeeper: 3.4.6 Kafka broker: 0.8.2-beta Kafka clients: 0.8.2-beta # Kafka boker settings (all other settings are default 0.8.2-beta settings) kafka.controlled.shutdown.enable: 'FALSE' kafka.auto.create.topics.enable: 'TRUE' kafka.num.partitions: 8 kafka.default.replication.factor: 1 kafka.rebalance.backoff.ms: 3000 kafka.rebalance.max.retries: 10 kafka.log.retention.minutes: 1200 kafka.delete.topic.enable: 'TRUE'
