Vinay, a "last access" timestamp would be much harder to implement, as it
would be a mutable property by nature. In contrast, the timestamp of a
topic creation is immutable and persisted (AFAIK).
I was really surprised to see that this information is not already
available through the Kafka Admin client. Any idea why? I would assume this
information to be valuable to a lot of things in the Kafka ecosystem.

I'm not too familiar with the process for suggesting enhancements to Kafka.
Would an addition like this require a KIP?

Regards,
Erik

On Mon, Oct 21, 2024 at 7:14 PM Vinay Bagare <vinay.bag...@gmail.com> wrote:

> I had been tracking this and my intention is to capture last access
> timestamp for identifying topic activity staleness,
> I feel we would probably benefit from having both Create/Last access
> Timestamp similar to what Erik is requesting (I hope I am understanding his
> requirement).
> For now, I am doing a hula dance (round about) by describing the topic and
> identifying its last activity (if anyone has better suggestions instead of
> this, please do let me know).
>
>
>
> Best,
> Vinay Bagare
>
> From: Erik Godding Boye <egb...@gmail.com>
> Date: Monday, October 21, 2024 at 10:17 AM
> To: users@kafka.apache.org <users@kafka.apache.org>
> Subject: Re: Get topic creationTimestamp through Admin Client (or Client)
> Hi Ömer,
>
> I don't disagree with you on the general principles for deletion, but let's
> keep focus on the question I posted. :-) And yes, I am aware of the
> LinkedIn state-machine for cleaning up metadata.
>
> > What do you mean by "this must be done external to " ?
>
> I mean that I want to do this with brokers as my only interface (Client
> and/or Admin Client). We are in the process of migrating from zookeeper to
> KRaft mode, and I don't want the burden of having to coordinate this
> migration into my toolbox. In addition, the clusters are insulated, so
> zookeepers/controllers are unavailable to this app.
>
> Wouldn't it make sense to include a creation timestamp in the topic
> metadata returned to an Admin Client  - if it is possible?
>
> Regards,
> Erik
>
> On Mon, Oct 21, 2024 at 4:24 PM Ömer Şiar Baysal <osiarbay...@gmail.com>
> wrote:
>
> > Hi Erik,
> >
> > What do you mean by "this must be done external to " ?
> >
> > If the cluster still uses ZK for metadata, you can construct a read-only
> > Zookeeper client to get zNode creation timestamp for the topic from ZK
> > node, which reflects the creation time.
> >
> > Also determining unused topics can be tricky if consumers come and go by
> > irregular basis, for example some batch jobs for BI and/or monthly
> > generated reports, etc.
> >
> > Check this blog post to see how LinkedIn does it.
> >
> >
> >
> https://engineering.linkedin.com/content/engineering/en-us/blog/2022/topicgc_how-linkedin-cleans-up-unused-metadata-for-its-kafka-clu
> >
> > IMHO hard deletes should be avoided for any kind of data platform.
> >
> > Kind regards,
> > OSB
> >
> >
> >
> >
> > On Sun, Oct 20, 2024, 14:24 Erik Godding Boye <egb...@gmail.com> wrote:
> >
> > > Hi Kafka experts,
> > >
> > > I wrote a small utility application to terminate (delete) unused topics
> > in
> > > our clusters. The application is hosted at
> > > https://github.com/statnett/k3a-topic-terminator (but this is not an
> > > attempt to promote the app; it's fresh ;-). We just got rid of 2816
> > topics
> > > in one of our clusters, and now we want to find even more garbage.
> > >
> > > So my question is: *Can the topic creation timestamp be obtained
> through
> > a
> > > Kafka Admin Client (or Client)?*
> > >
> > > I think the answer is NO, but would it be possible to implement such a
> > > feature? As a user, I would expect to find this information through the
> > > TopicDescription
> > > <
> > >
> >
> https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/admin/TopicDescription.html
> > > >
> > > class.As far as I understand, this information is available in the
> > cluster
> > > - either within Zookeepers and/or on the __cluster_metadata topic. I
> have
> > > tried to set up a consumer towards the metadata topic, but no luck so
> > far.
> > > I am very interested in potential workarounds until this feature is
> > > eventually implemented! In my case, this *must* be done external to the
> > > brokers/controllers/zookeepers. Running a shell script or CLI tool is
> > not a
> > > feasible option (in this case).
> > >
> > > Regards.
> > > Erik
> > >
> >
>

Reply via email to