Hi Chia-Ping, > Pardon me, what is the purpose of monitoring __cluster_metadata? We run one of our Kafka clusters on Kubernetes, where the IP address of the pods running the broker changes every time the pod is restarted for any reason. So we need to see as brokers are stopped and restarted, so that we get it's new IP and can update it on our service mesh configuration. We probably run Kafka with the mesh together in an unusual way. It worked well so far having Zookeeper, we just needed to watch some of the znodes to get the updates. But now as we'll soon need to say goodbye to Zookeeper, we need another way to quickly discover the new/restarted brokers and some of its configuration.
The metadata topic would be all we need and we could easily filter out the information we don't need. I see other options as well to accomplish our needs, but reading the metadata topic would be probably the simplest and easiest. Cheers, Peter On Thu, 17 Apr 2025 at 20:12, Chia-Ping Tsai <chia7...@gmail.com> wrote: > hi Peter > > Pardon me, what is the purpose of monitoring __cluster_metadata? Are you > looking to observe topic changes or other cluster modifications? > > As brokers continuously send heartbeat requests, __cluster_metadata will > keep "updating" even if there are no topic changes. > Best, > Chia-Ping > > > Péter Sinóros-Szabó <peter.sinoros-sz...@wise.com.invalid> 於 2025年4月18日 週五 > 上午1:30寫道： > > > Hi, > > > > I'm looking for a way to consume the cluster metadata updates when using > > KRaft. I thought I'll just consume the __cluster_metadata topic, but as I > > see that's not possible, at least I didn't manage to do that with the > > console consumer. > > > > I know we have the dump tool or the metadata shell to query the cluster > > metadata, but I'd need something that I can stream, so I'm notified as > soon > > as there's a change. We used to watch the znodes in Zookeeper for that > > reason, but now I'm looking for an alternative solution with KRaft. > > > > Is there a way to consume these cluster metadata updates? If it's not > > possible at the moment is it maybe possible to enable it to read in a > > version that still supports Zookeeper? > > > > Thanks, > > Peter > > >
