Vinay, a "last access" timestamp would be much harder to implement, as it would be a mutable property by nature. In contrast, the timestamp of a topic creation is immutable and persisted (AFAIK). I was really surprised to see that this information is not already available through the Kafka Admin client. Any idea why? I would assume this information to be valuable to a lot of things in the Kafka ecosystem.
I'm not too familiar with the process for suggesting enhancements to Kafka. Would an addition like this require a KIP? Regards, Erik On Mon, Oct 21, 2024 at 7:14 PM Vinay Bagare <vinay.bag...@gmail.com> wrote: > I had been tracking this and my intention is to capture last access > timestamp for identifying topic activity staleness, > I feel we would probably benefit from having both Create/Last access > Timestamp similar to what Erik is requesting (I hope I am understanding his > requirement). > For now, I am doing a hula dance (round about) by describing the topic and > identifying its last activity (if anyone has better suggestions instead of > this, please do let me know). > > > > Best, > Vinay Bagare > > From: Erik Godding Boye <egb...@gmail.com> > Date: Monday, October 21, 2024 at 10:17 AM > To: users@kafka.apache.org <users@kafka.apache.org> > Subject: Re: Get topic creationTimestamp through Admin Client (or Client) > Hi Ömer, > > I don't disagree with you on the general principles for deletion, but let's > keep focus on the question I posted. :-) And yes, I am aware of the > LinkedIn state-machine for cleaning up metadata. > > > What do you mean by "this must be done external to " ? > > I mean that I want to do this with brokers as my only interface (Client > and/or Admin Client). We are in the process of migrating from zookeeper to > KRaft mode, and I don't want the burden of having to coordinate this > migration into my toolbox. In addition, the clusters are insulated, so > zookeepers/controllers are unavailable to this app. > > Wouldn't it make sense to include a creation timestamp in the topic > metadata returned to an Admin Client - if it is possible? > > Regards, > Erik > > On Mon, Oct 21, 2024 at 4:24 PM Ömer Şiar Baysal <osiarbay...@gmail.com> > wrote: > > > Hi Erik, > > > > What do you mean by "this must be done external to " ? > > > > If the cluster still uses ZK for metadata, you can construct a read-only > > Zookeeper client to get zNode creation timestamp for the topic from ZK > > node, which reflects the creation time. > > > > Also determining unused topics can be tricky if consumers come and go by > > irregular basis, for example some batch jobs for BI and/or monthly > > generated reports, etc. > > > > Check this blog post to see how LinkedIn does it. > > > > > > > https://engineering.linkedin.com/content/engineering/en-us/blog/2022/topicgc_how-linkedin-cleans-up-unused-metadata-for-its-kafka-clu > > > > IMHO hard deletes should be avoided for any kind of data platform. > > > > Kind regards, > > OSB > > > > > > > > > > On Sun, Oct 20, 2024, 14:24 Erik Godding Boye <egb...@gmail.com> wrote: > > > > > Hi Kafka experts, > > > > > > I wrote a small utility application to terminate (delete) unused topics > > in > > > our clusters. The application is hosted at > > > https://github.com/statnett/k3a-topic-terminator (but this is not an > > > attempt to promote the app; it's fresh ;-). We just got rid of 2816 > > topics > > > in one of our clusters, and now we want to find even more garbage. > > > > > > So my question is: *Can the topic creation timestamp be obtained > through > > a > > > Kafka Admin Client (or Client)?* > > > > > > I think the answer is NO, but would it be possible to implement such a > > > feature? As a user, I would expect to find this information through the > > > TopicDescription > > > < > > > > > > https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/admin/TopicDescription.html > > > > > > > class.As far as I understand, this information is available in the > > cluster > > > - either within Zookeepers and/or on the __cluster_metadata topic. I > have > > > tried to set up a consumer towards the metadata topic, but no luck so > > far. > > > I am very interested in potential workarounds until this feature is > > > eventually implemented! In my case, this *must* be done external to the > > > brokers/controllers/zookeepers. Running a shell script or CLI tool is > > not a > > > feasible option (in this case). > > > > > > Regards. > > > Erik > > > > > >
