Thanks, for standalone consumers I do not have any useful monitoring tool on top of my head, maybe this KIP looks promising to you, but it's still WIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability
Also regarding ACL setup, I think that's also a good idea, and you should be able to restrict data reads https://docs.confluent.io/platform/current/kafka/authorization.html#operations or I'm missing some context here. On Thu, Oct 7, 2021 at 12:44 PM Murilo Tavares <murilo...@gmail.com> wrote: > Hi Boyang > Thanks for your response. > Yes, I'm talking about that framework. > goka has a few different things. What they call `views` are similar to > Global KTables in KafkaStreams. It's a full snapshot of the Kafka topic, > held in memory. So every time you spin up your service, the goka view will > read the topic from beginning, thus no need to keep track of consumer > offset or consumer groups. > Thanks > Murilo > > On Thu, 7 Oct 2021 at 15:33, Boyang Chen <reluctanthero...@gmail.com> > wrote: > > > Hey Murilo, could you explain what you mean by `goka views`? If you are > > talking about https://github.com/lovoo/goka, they should use consumer > > groups as well IIUC. > > > > Boyang > > > > On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares <murilo...@gmail.com> > > wrote: > > > > > Hi. Looking for some insights here. > > > We use Kafka at a large scale, and have lots of microservices using > Kafka > > > for all sorts of things. > > > Our biggest challenge nowadays is to track which topics are used and > > which > > > are not. > > > I have considered looking at consumer groups to identify which > > applications > > > consume from each topic, but then I realized we have lots of > > microservices > > > that do not use consumer groups (lots of goka views on top of Kafka > > > topics). > > > > > > Is there any architecture solution that can help track consumers of a > > Kafka > > > topic? > > > Thought that I could use ACL to restrict the creation of Kafka Topics > and > > > use a gatekeeper service to centralize and log who creates the topics, > > but > > > still that does not block anyone from consuming from them. > > > > > > Any thoughts on that? > > > Thanks > > > Murilo > > > > > >
