I had been tracking this and my intention is to capture last access timestamp 
for identifying topic activity staleness,
I feel we would probably benefit from having both Create/Last access Timestamp 
similar to what Erik is requesting (I hope I am understanding his requirement).
For now, I am doing a hula dance (round about) by describing the topic and 
identifying its last activity (if anyone has better suggestions instead of 
this, please do let me know).



Best,
Vinay Bagare

From: Erik Godding Boye <egb...@gmail.com>
Date: Monday, October 21, 2024 at 10:17 AM
To: users@kafka.apache.org <users@kafka.apache.org>
Subject: Re: Get topic creationTimestamp through Admin Client (or Client)
Hi Ömer,

I don't disagree with you on the general principles for deletion, but let's
keep focus on the question I posted. :-) And yes, I am aware of the
LinkedIn state-machine for cleaning up metadata.

> What do you mean by "this must be done external to " ?

I mean that I want to do this with brokers as my only interface (Client
and/or Admin Client). We are in the process of migrating from zookeeper to
KRaft mode, and I don't want the burden of having to coordinate this
migration into my toolbox. In addition, the clusters are insulated, so
zookeepers/controllers are unavailable to this app.

Wouldn't it make sense to include a creation timestamp in the topic
metadata returned to an Admin Client  - if it is possible?

Regards,
Erik

On Mon, Oct 21, 2024 at 4:24 PM Ömer Şiar Baysal <osiarbay...@gmail.com>
wrote:

> Hi Erik,
>
> What do you mean by "this must be done external to " ?
>
> If the cluster still uses ZK for metadata, you can construct a read-only
> Zookeeper client to get zNode creation timestamp for the topic from ZK
> node, which reflects the creation time.
>
> Also determining unused topics can be tricky if consumers come and go by
> irregular basis, for example some batch jobs for BI and/or monthly
> generated reports, etc.
>
> Check this blog post to see how LinkedIn does it.
>
>
> https://engineering.linkedin.com/content/engineering/en-us/blog/2022/topicgc_how-linkedin-cleans-up-unused-metadata-for-its-kafka-clu
>
> IMHO hard deletes should be avoided for any kind of data platform.
>
> Kind regards,
> OSB
>
>
>
>
> On Sun, Oct 20, 2024, 14:24 Erik Godding Boye <egb...@gmail.com> wrote:
>
> > Hi Kafka experts,
> >
> > I wrote a small utility application to terminate (delete) unused topics
> in
> > our clusters. The application is hosted at
> > https://github.com/statnett/k3a-topic-terminator (but this is not an
> > attempt to promote the app; it's fresh ;-). We just got rid of 2816
> topics
> > in one of our clusters, and now we want to find even more garbage.
> >
> > So my question is: *Can the topic creation timestamp be obtained through
> a
> > Kafka Admin Client (or Client)?*
> >
> > I think the answer is NO, but would it be possible to implement such a
> > feature? As a user, I would expect to find this information through the
> > TopicDescription
> > <
> >
> https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/admin/TopicDescription.html
> > >
> > class.As far as I understand, this information is available in the
> cluster
> > - either within Zookeepers and/or on the __cluster_metadata topic. I have
> > tried to set up a consumer towards the metadata topic, but no luck so
> far.
> > I am very interested in potential workarounds until this feature is
> > eventually implemented! In my case, this *must* be done external to the
> > brokers/controllers/zookeepers. Running a shell script or CLI tool is
> not a
> > feasible option (in this case).
> >
> > Regards.
> > Erik
> >
>

Reply via email to