Hi Abhishek, as I told you before, topic is getting created only because of consumer polling. The only way is to stop consumer, remove topic name from the consumer, delete topic.
Here I assume your consumer knows which topic to read from( by static topic name) If not, I dont know how your consumer knows excat topic name where to read from? In my case we had static topic names given in consumer config. So removed those before topic deletion. Alternatively you can set auto create false by rolling restart all your brokers if number of brokers and not much. Tbat will be quick fix On Sun, 18 Feb 2024 at 6:07 PM, Abhishek Singla <abhisheksingla...@gmail.com> wrote: > Hi Megh, > > Thanks for the detailed explanation. I tested it out and below are the > observations. > > When the delete topic command is issued, the topic is marked for deletion > and it is deleted within the next 1 minute. In this 1 minute interval, the > consumer offsets are there in __consumer_offsets topic but the describe > group command fails with below error. > *Error: Executing consumer group command failed due to > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This > server does not host this topic-partition.* > > Now, within that one minute if consumer is running then the topic gets > created again and offset is reset to 0 and if consumer is not running then > offsets are cleared automatically from __consumer_offsets topic and topic > is not created upon resuming consumers. > > kafka-consumer-groups.sh --delete-offsets will expedite the offset > clearance bit but it also requires consumers to be stopped. One more thing > I noticed is --consumer-property allow.auto.create.topics=false does not > restrict topic creation in this particular scenario. > > Is there any other way I can delete the topics without stopping consumers? > > Regards, > Abhishek Singla > > > > On Sun, Feb 18, 2024 at 3:18 PM megh vidani <vidanimeg...@gmail.com> > wrote: > > > Hi Abhishek, > > > > The consumer offsets get stored in the __consumer_offsets topic by the > > kafka brokers and don't get automatically deleted when you delete a > topic. > > If you have auto.create.topics.enable set to true in your broker configs, > > th consumer automatically creates the topic again. > > > > You will have to delete the offsets for that topic using the > > kafka-consumer-groups.sh for your consumer group in order to remove the > > topic offsets for that consumer group. > > > > Also, you can disable automatic creation of topic in the kafka brokers > > config using auto.create.topics.enable property. > > > > Thanks, > > Megh > > > > > > On Sun, Feb 18, 2024, 15:09 Abhishek Singla <abhisheksingla...@gmail.com > > > > wrote: > > > > > We only delete topics which does not have any active producers. > > > > > > Nowhere in our consumer config we explicitly specify topic name, these > > are > > > subscribed based on pattern. How do I remove topic from consumer > config? > > > > > > On Sun, 18 Feb 2024 at 2:51 PM, sunil chaudhari < > > > sunilmchaudhar...@gmail.com> > > > wrote: > > > > > > > correcte missig thing. > > > > You have to remove that topic from producer as well as consumer > config > > > > both. > > > > > > > > On Sun, 18 Feb 2024 at 2:49 PM, sunil chaudhari < > > > > sunilmchaudhar...@gmail.com> > > > > wrote: > > > > > > > > > You have to remove that topic from consumer config. > > > > > restart consumer. > > > > > the wait for some time. > > > > > Then delete topic. > > > > > this time it wont create again. > > > > > > > > > > On Sun, 18 Feb 2024 at 1:07 PM, Abhishek Singla < > > > > > abhisheksingla...@gmail.com> wrote: > > > > > > > > > >> Hi Team, > > > > >> > > > > >> Kafka version: 2_2.12-2.6.0 > > > > >> Zookeeper version: 3.4.14 > > > > >> Java version: 1.8.0_301 > > > > >> > > > > >> Kafka Subscribe Pattern API is used to Subscribe to all topics > > > matching > > > > >> specified pattern to get dynamically assigned partitions. New > topics > > > of > > > > >> specified pattern get created with time. > > > > >> > > > > >> > KafkaConsumer<K, V> subscribe(Pattern pattern, > > > > >> Handler<AsyncResult<Void>> > > > > >> completionHandler); > > > > >> > > > > >> The issue arises when we try to delete the topic. After the delete > > > topic > > > > >> command is issued, the topic is deleted successfully. However, it > > gets > > > > >> auto-created again within 5 mins. Broker offsets are reset to > zero, > > > new > > > > >> topic partitions could be on same broker nodes or different. > > > > >> > > > > >> Below are some of the configs used: > > > > >> > > > > >> > > > > >> *Server Configs:* > > > > >> delete.topic.enable=true > > > > >> auto.create.topics.enable: true > > > > >> > > > > >> *Consumer Configs* > > > > >> allow.auto.create.topics=false > > > > >> > > > > >> Could someone help us out in understanding what might be the root > > > cause > > > > of > > > > >> this issue? > > > > >> > > > > >> Regards, > > > > >> Abhishek Singla > > > > >> > > > > > > > > > > > > > > >
