I had been tracking this and my intention is to capture last access timestamp for identifying topic activity staleness, I feel we would probably benefit from having both Create/Last access Timestamp similar to what Erik is requesting (I hope I am understanding his requirement). For now, I am doing a hula dance (round about) by describing the topic and identifying its last activity (if anyone has better suggestions instead of this, please do let me know).
Best, Vinay Bagare From: Erik Godding Boye <egb...@gmail.com> Date: Monday, October 21, 2024 at 10:17 AM To: users@kafka.apache.org <users@kafka.apache.org> Subject: Re: Get topic creationTimestamp through Admin Client (or Client) Hi Ömer, I don't disagree with you on the general principles for deletion, but let's keep focus on the question I posted. :-) And yes, I am aware of the LinkedIn state-machine for cleaning up metadata. > What do you mean by "this must be done external to " ? I mean that I want to do this with brokers as my only interface (Client and/or Admin Client). We are in the process of migrating from zookeeper to KRaft mode, and I don't want the burden of having to coordinate this migration into my toolbox. In addition, the clusters are insulated, so zookeepers/controllers are unavailable to this app. Wouldn't it make sense to include a creation timestamp in the topic metadata returned to an Admin Client - if it is possible? Regards, Erik On Mon, Oct 21, 2024 at 4:24 PM Ömer Şiar Baysal <osiarbay...@gmail.com> wrote: > Hi Erik, > > What do you mean by "this must be done external to " ? > > If the cluster still uses ZK for metadata, you can construct a read-only > Zookeeper client to get zNode creation timestamp for the topic from ZK > node, which reflects the creation time. > > Also determining unused topics can be tricky if consumers come and go by > irregular basis, for example some batch jobs for BI and/or monthly > generated reports, etc. > > Check this blog post to see how LinkedIn does it. > > > https://engineering.linkedin.com/content/engineering/en-us/blog/2022/topicgc_how-linkedin-cleans-up-unused-metadata-for-its-kafka-clu > > IMHO hard deletes should be avoided for any kind of data platform. > > Kind regards, > OSB > > > > > On Sun, Oct 20, 2024, 14:24 Erik Godding Boye <egb...@gmail.com> wrote: > > > Hi Kafka experts, > > > > I wrote a small utility application to terminate (delete) unused topics > in > > our clusters. The application is hosted at > > https://github.com/statnett/k3a-topic-terminator (but this is not an > > attempt to promote the app; it's fresh ;-). We just got rid of 2816 > topics > > in one of our clusters, and now we want to find even more garbage. > > > > So my question is: *Can the topic creation timestamp be obtained through > a > > Kafka Admin Client (or Client)?* > > > > I think the answer is NO, but would it be possible to implement such a > > feature? As a user, I would expect to find this information through the > > TopicDescription > > < > > > https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/admin/TopicDescription.html > > > > > class.As far as I understand, this information is available in the > cluster > > - either within Zookeepers and/or on the __cluster_metadata topic. I have > > tried to set up a consumer towards the metadata topic, but no luck so > far. > > I am very interested in potential workarounds until this feature is > > eventually implemented! In my case, this *must* be done external to the > > brokers/controllers/zookeepers. Running a shell script or CLI tool is > not a > > feasible option (in this case). > > > > Regards. > > Erik > > >
