Issue is now resolved by upgrading zookeeper version from 3.4.14 to 3.8.0 On Sun, Feb 18, 2024 at 6:15 PM sunil chaudhari <sunilmchaudhar...@gmail.com> wrote:
> Hi Abhishek, > as I told you before, topic is getting created only because of consumer > polling. > The only way is to stop consumer, remove topic name from the consumer, > delete topic. > > Here I assume your consumer knows which topic to read from( by static topic > name) > If not, I dont know how your consumer knows excat topic name where to read > from? > > In my case we had static topic names given in consumer config. So removed > those before topic deletion. > > Alternatively you can set auto create false by rolling restart all your > brokers if number of brokers and not much. > Tbat will be quick fix > > > > On Sun, 18 Feb 2024 at 6:07 PM, Abhishek Singla < > abhisheksingla...@gmail.com> > wrote: > > > Hi Megh, > > > > Thanks for the detailed explanation. I tested it out and below are the > > observations. > > > > When the delete topic command is issued, the topic is marked for deletion > > and it is deleted within the next 1 minute. In this 1 minute interval, > the > > consumer offsets are there in __consumer_offsets topic but the describe > > group command fails with below error. > > *Error: Executing consumer group command failed due to > > org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This > > server does not host this topic-partition.* > > > > Now, within that one minute if consumer is running then the topic gets > > created again and offset is reset to 0 and if consumer is not running > then > > offsets are cleared automatically from __consumer_offsets topic and topic > > is not created upon resuming consumers. > > > > kafka-consumer-groups.sh --delete-offsets will expedite the offset > > clearance bit but it also requires consumers to be stopped. One more > thing > > I noticed is --consumer-property allow.auto.create.topics=false does not > > restrict topic creation in this particular scenario. > > > > Is there any other way I can delete the topics without stopping > consumers? > > > > Regards, > > Abhishek Singla > > > > > > > > On Sun, Feb 18, 2024 at 3:18 PM megh vidani <vidanimeg...@gmail.com> > > wrote: > > > > > Hi Abhishek, > > > > > > The consumer offsets get stored in the __consumer_offsets topic by the > > > kafka brokers and don't get automatically deleted when you delete a > > topic. > > > If you have auto.create.topics.enable set to true in your broker > configs, > > > th consumer automatically creates the topic again. > > > > > > You will have to delete the offsets for that topic using the > > > kafka-consumer-groups.sh for your consumer group in order to remove the > > > topic offsets for that consumer group. > > > > > > Also, you can disable automatic creation of topic in the kafka brokers > > > config using auto.create.topics.enable property. > > > > > > Thanks, > > > Megh > > > > > > > > > On Sun, Feb 18, 2024, 15:09 Abhishek Singla < > abhisheksingla...@gmail.com > > > > > > wrote: > > > > > > > We only delete topics which does not have any active producers. > > > > > > > > Nowhere in our consumer config we explicitly specify topic name, > these > > > are > > > > subscribed based on pattern. How do I remove topic from consumer > > config? > > > > > > > > On Sun, 18 Feb 2024 at 2:51 PM, sunil chaudhari < > > > > sunilmchaudhar...@gmail.com> > > > > wrote: > > > > > > > > > correcte missig thing. > > > > > You have to remove that topic from producer as well as consumer > > config > > > > > both. > > > > > > > > > > On Sun, 18 Feb 2024 at 2:49 PM, sunil chaudhari < > > > > > sunilmchaudhar...@gmail.com> > > > > > wrote: > > > > > > > > > > > You have to remove that topic from consumer config. > > > > > > restart consumer. > > > > > > the wait for some time. > > > > > > Then delete topic. > > > > > > this time it wont create again. > > > > > > > > > > > > On Sun, 18 Feb 2024 at 1:07 PM, Abhishek Singla < > > > > > > abhisheksingla...@gmail.com> wrote: > > > > > > > > > > > >> Hi Team, > > > > > >> > > > > > >> Kafka version: 2_2.12-2.6.0 > > > > > >> Zookeeper version: 3.4.14 > > > > > >> Java version: 1.8.0_301 > > > > > >> > > > > > >> Kafka Subscribe Pattern API is used to Subscribe to all topics > > > > matching > > > > > >> specified pattern to get dynamically assigned partitions. New > > topics > > > > of > > > > > >> specified pattern get created with time. > > > > > >> > > > > > >> > KafkaConsumer<K, V> subscribe(Pattern pattern, > > > > > >> Handler<AsyncResult<Void>> > > > > > >> completionHandler); > > > > > >> > > > > > >> The issue arises when we try to delete the topic. After the > delete > > > > topic > > > > > >> command is issued, the topic is deleted successfully. However, > it > > > gets > > > > > >> auto-created again within 5 mins. Broker offsets are reset to > > zero, > > > > new > > > > > >> topic partitions could be on same broker nodes or different. > > > > > >> > > > > > >> Below are some of the configs used: > > > > > >> > > > > > >> > > > > > >> *Server Configs:* > > > > > >> delete.topic.enable=true > > > > > >> auto.create.topics.enable: true > > > > > >> > > > > > >> *Consumer Configs* > > > > > >> allow.auto.create.topics=false > > > > > >> > > > > > >> Could someone help us out in understanding what might be the > root > > > > cause > > > > > of > > > > > >> this issue? > > > > > >> > > > > > >> Regards, > > > > > >> Abhishek Singla > > > > > >> > > > > > > > > > > > > > > > > > > > > >
