Hi Megh, Thanks for the detailed explanation. I tested it out and below are the observations.
When the delete topic command is issued, the topic is marked for deletion and it is deleted within the next 1 minute. In this 1 minute interval, the consumer offsets are there in __consumer_offsets topic but the describe group command fails with below error. *Error: Executing consumer group command failed due to org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.* Now, within that one minute if consumer is running then the topic gets created again and offset is reset to 0 and if consumer is not running then offsets are cleared automatically from __consumer_offsets topic and topic is not created upon resuming consumers. kafka-consumer-groups.sh --delete-offsets will expedite the offset clearance bit but it also requires consumers to be stopped. One more thing I noticed is --consumer-property allow.auto.create.topics=false does not restrict topic creation in this particular scenario. Is there any other way I can delete the topics without stopping consumers? Regards, Abhishek Singla On Sun, Feb 18, 2024 at 3:18 PM megh vidani <vidanimeg...@gmail.com> wrote: > Hi Abhishek, > > The consumer offsets get stored in the __consumer_offsets topic by the > kafka brokers and don't get automatically deleted when you delete a topic. > If you have auto.create.topics.enable set to true in your broker configs, > th consumer automatically creates the topic again. > > You will have to delete the offsets for that topic using the > kafka-consumer-groups.sh for your consumer group in order to remove the > topic offsets for that consumer group. > > Also, you can disable automatic creation of topic in the kafka brokers > config using auto.create.topics.enable property. > > Thanks, > Megh > > > On Sun, Feb 18, 2024, 15:09 Abhishek Singla <abhisheksingla...@gmail.com> > wrote: > > > We only delete topics which does not have any active producers. > > > > Nowhere in our consumer config we explicitly specify topic name, these > are > > subscribed based on pattern. How do I remove topic from consumer config? > > > > On Sun, 18 Feb 2024 at 2:51 PM, sunil chaudhari < > > sunilmchaudhar...@gmail.com> > > wrote: > > > > > correcte missig thing. > > > You have to remove that topic from producer as well as consumer config > > > both. > > > > > > On Sun, 18 Feb 2024 at 2:49 PM, sunil chaudhari < > > > sunilmchaudhar...@gmail.com> > > > wrote: > > > > > > > You have to remove that topic from consumer config. > > > > restart consumer. > > > > the wait for some time. > > > > Then delete topic. > > > > this time it wont create again. > > > > > > > > On Sun, 18 Feb 2024 at 1:07 PM, Abhishek Singla < > > > > abhisheksingla...@gmail.com> wrote: > > > > > > > >> Hi Team, > > > >> > > > >> Kafka version: 2_2.12-2.6.0 > > > >> Zookeeper version: 3.4.14 > > > >> Java version: 1.8.0_301 > > > >> > > > >> Kafka Subscribe Pattern API is used to Subscribe to all topics > > matching > > > >> specified pattern to get dynamically assigned partitions. New topics > > of > > > >> specified pattern get created with time. > > > >> > > > >> > KafkaConsumer<K, V> subscribe(Pattern pattern, > > > >> Handler<AsyncResult<Void>> > > > >> completionHandler); > > > >> > > > >> The issue arises when we try to delete the topic. After the delete > > topic > > > >> command is issued, the topic is deleted successfully. However, it > gets > > > >> auto-created again within 5 mins. Broker offsets are reset to zero, > > new > > > >> topic partitions could be on same broker nodes or different. > > > >> > > > >> Below are some of the configs used: > > > >> > > > >> > > > >> *Server Configs:* > > > >> delete.topic.enable=true > > > >> auto.create.topics.enable: true > > > >> > > > >> *Consumer Configs* > > > >> allow.auto.create.topics=false > > > >> > > > >> Could someone help us out in understanding what might be the root > > cause > > > of > > > >> this issue? > > > >> > > > >> Regards, > > > >> Abhishek Singla > > > >> > > > > > > > > > >
