> > >> I have since stopped the container so I cannot say if > LAX1-GRIFFIN-r45-1423000088317 was one of the topic in "marked for > deletion" forever. However, there were many topics (at least 10 of them) > that were perennially in "marked for deletion" state. > > I have the setup to recreate the issue in case the logs are not sufficient.
Sumit > Sumit > > > > > >> -Harsha >> >> On Tue, Feb 3, 2015, at 09:19 PM, Harsha wrote: >> > you are probably handling it but there is a case where you call >> > deleteTopic and kafka goes through delete topic process but your >> > consumer is running probably made a TopicMetadataRequest for the same >> > topic which can re-create the topic with the default num.partitions and >> > replication.factor. Did you try stopping the consumer first and issue >> > the topic delete. >> > -Harsha >> > >> > On Tue, Feb 3, 2015, at 08:37 PM, Sumit Rangwala wrote: >> > > On Tue, Feb 3, 2015 at 6:48 PM, Harsha <ka...@harsha.io> wrote: >> > > >> > > > Sumit, >> > > > lets say you are deleting a older topic "test1" do you have >> any >> > > > consumers running simultaneously for the topic "test1" while >> > > > deletion of topic going on. >> > > > >> > > >> > > Yes it is the case. However, after a small period of time (say few >> > > minutes) >> > > there won't be any consumer running for the deleted topic. >> > > >> > > >> > > Sumit >> > > >> > > >> > > >> > > >> > > > -Harsha >> > > > >> > > > On Tue, Feb 3, 2015, at 06:17 PM, Joel Koshy wrote: >> > > > > Thanks for the logs - will take a look tomorrow unless someone >> else >> > > > > gets a chance to get to it today. >> > > > > >> > > > > Joel >> > > > > >> > > > > On Tue, Feb 03, 2015 at 04:11:57PM -0800, Sumit Rangwala wrote: >> > > > > > On Tue, Feb 3, 2015 at 3:37 PM, Joel Koshy <jjkosh...@gmail.com >> > >> > > > wrote: >> > > > > > >> > > > > > > Hey Sumit, >> > > > > > > >> > > > > > > I thought you would be providing the actual steps to >> reproduce :) >> > > > > > > >> > > > > > >> > > > > > I want to but some proprietary code prevents me to do it. >> > > > > > >> > > > > > >> > > > > > > Nevertheless, can you get all the relevant logs: state change >> logs >> > > > and >> > > > > > > controller logs at the very least and if possible server logs >> and >> > > > send >> > > > > > > those over? >> > > > > > > >> > > > > > >> > > > > > Here are all the logs you requested (there are three brokers in >> my >> > > > setup >> > > > > > k1, k2, k3): http://d.pr/f/1kprY/2quHBRRT (Gmail has issue >> with the >> > > > file) >> > > > > > >> > > > > > >> > > > > > Sumit >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > >> > > > > > > Joel >> > > > > > > >> > > > > > > On Tue, Feb 03, 2015 at 03:27:43PM -0800, Sumit Rangwala >> wrote: >> > > > > > > > In my setup kafka brokers are set for auto topic creation. >> In the >> > > > > > > scenario >> > > > > > > > below a node informs other nodes (currently 5 in total) >> about a >> > > > number >> > > > > > > of >> > > > > > > > new (non-existent) topics, and all the nodes almost >> > > > simultaneously open >> > > > > > > a >> > > > > > > > consumer for each of those topics. Sometime later another >> node >> > > > informs >> > > > > > > all >> > > > > > > > other nodes of a new list of topics and each node, if they >> find >> > > > that an >> > > > > > > > older topic exists in kafka, goes ahead and deletes the >> older >> > > > topic. >> > > > > > > What I >> > > > > > > > have found is that many of the topics stay in the "marked >> for >> > > > deletion" >> > > > > > > > state forever. >> > > > > > > > >> > > > > > > > >> > > > > > > > I get the list of topics using >> ZkUtils.getAllTopics(zkClient) and >> > > > delete >> > > > > > > > topics using AdminUtils.deleteTopic(zkClient, topic). Since >> many >> > > > nodes >> > > > > > > > might try to delete the same topic at the same time I do >> > > > > > > > see ZkNodeExistsException while deleting the topic, which I >> catch >> > > > an >> > > > > > > > ignore. (e.g., >> > > > org.apache.zookeeper.KeeperException$NodeExistsException: >> > > > > > > > KeeperErrorCode = NodeExists for >> > > > > > > > /admin/delete_topics/LAX1-GRIFFIN-r13-1423001701601) >> > > > > > > > >> > > > > > > > # State of one deleted topic on kafka brokers: >> > > > > > > > Topic:LAX1-GRIFFIN-r13-1423001701601 PartitionCount:8 >> > > > ReplicationFactor:1 >> > > > > > > > Configs: >> > > > > > > > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 0 Leader: >> -1 >> > > > Replicas: >> > > > > > > > 49558 Isr: >> > > > > > > > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 1 Leader: >> -1 >> > > > Replicas: >> > > > > > > > 49554 Isr: >> > > > > > > > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 2 Leader: >> -1 >> > > > Replicas: >> > > > > > > > 49557 Isr: >> > > > > > > > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 3 Leader: >> -1 >> > > > Replicas: >> > > > > > > > 49558 Isr: >> > > > > > > > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 4 Leader: >> -1 >> > > > Replicas: >> > > > > > > > 49554 Isr: >> > > > > > > > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 5 Leader: >> -1 >> > > > Replicas: >> > > > > > > > 49557 Isr: >> > > > > > > > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 6 Leader: >> -1 >> > > > Replicas: >> > > > > > > > 49558 Isr: >> > > > > > > > Topic: LAX1-GRIFFIN-r13-1423001701601 Partition: 7 Leader: >> -1 >> > > > Replicas: >> > > > > > > > 49554 Isr: >> > > > > > > > >> > > > > > > > >> > > > > > > > # Controller log says >> > > > > > > > >> > > > > > > > [2015-02-03 22:59:03,399] INFO [delete-topics-thread-49554], >> > > > Deletion for >> > > > > > > > replicas 49557,49554,49558 for partition >> > > > > > > > >> > > > > > > >> > > > >> [LAX1-GRIFFIN-r13-1423001701601,0],[LAX1-GRIFFIN-r13-1423001701601,6],[LAX1-GRIFFIN-r13-1423001701601,5],[LAX1-GRIFFIN-r13-1423001701601,3],[LAX1-GRIFFIN-r13-1423001701601,7],[LAX1-GRIFFIN-r13-1423001701601,1],[LAX1-GRIFFIN-r13-1423001701601,4],[LAX1-GRIFFIN-r13-1423001701601,2] >> > > > > > > > of topic LAX1-GRIFFIN-r13-1423001701601 in progress >> > > > > > > > (kafka.controller.TopicDeletionManager$DeleteTopicsThread) >> > > > > > > > >> > > > > > > > current time: Tue Feb 3 23:20:58 UTC 2015 >> > > > > > > > >> > > > > > > > >> > > > > > > > Since I don't know the delete topic algorithm, I am not >> sure why >> > > > sure >> > > > > > > these >> > > > > > > > topics are not garbage collected. I do have the complete >> setup >> > > > running in >> > > > > > > > docker right now on my local box so please let me know if >> any more >> > > > info >> > > > > > > is >> > > > > > > > required to troubleshoot this issue. >> > > > > > > > >> > > > > > > > >> > > > > > > > Furthermore, does ZkUtils.getAllTopics(zkClient) return >> "marked for >> > > > > > > > deletion" topic as well. If so, is there a easy way to get >> a list >> > > > of >> > > > > > > active >> > > > > > > > topics (other than looking at all the topics in >> > > > /admin/delete_topics/ and >> > > > > > > > taking a set difference with the topics returned >> > > > > > > > by ZkUtils.getAllTopics(zkClient) ) >> > > > > > > > >> > > > > > > > Sumit >> > > > > > > > (More setup info below) >> > > > > > > > >> > > > > > > > >> > > > > > > > Setup >> > > > > > > > -------- >> > > > > > > > Zookeeper: 3.4.6 >> > > > > > > > Kafka broker: 0.8.2-beta >> > > > > > > > Kafka clients: 0.8.2-beta >> > > > > > > > >> > > > > > > > # Kafka boker settings (all other settings are default >> 0.8.2-beta >> > > > > > > settings) >> > > > > > > > kafka.controlled.shutdown.enable: 'FALSE' >> > > > > > > > kafka.auto.create.topics.enable: 'TRUE' >> > > > > > > > kafka.num.partitions: 8 >> > > > > > > > kafka.default.replication.factor: 1 >> > > > > > > > kafka.rebalance.backoff.ms: 3000 >> > > > > > > > kafka.rebalance.max.retries: 10 >> > > > > > > > kafka.log.retention.minutes: 1200 >> > > > > > > > kafka.delete.topic.enable: 'TRUE' >> > > > > > > >> > > > > > > >> > > > > >> > > > >> > >
