Oh cool. I had the wrong impression that ACL would not control read/writes, but just topic operations like CREATE and DELETE. Thanks for pointing that out.
On Thu, 7 Oct 2021 at 16:09, Boyang Chen <reluctanthero...@gmail.com> wrote: > Thanks, for standalone consumers I do not have any useful monitoring tool > on top of my head, maybe this KIP looks promising to you, but it's still > WIP > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability > > Also regarding ACL setup, I think that's also a good idea, and you should > be able to restrict data reads > > https://docs.confluent.io/platform/current/kafka/authorization.html#operations > or I'm missing some context here. > > On Thu, Oct 7, 2021 at 12:44 PM Murilo Tavares <murilo...@gmail.com> > wrote: > > > Hi Boyang > > Thanks for your response. > > Yes, I'm talking about that framework. > > goka has a few different things. What they call `views` are similar to > > Global KTables in KafkaStreams. It's a full snapshot of the Kafka topic, > > held in memory. So every time you spin up your service, the goka view > will > > read the topic from beginning, thus no need to keep track of consumer > > offset or consumer groups. > > Thanks > > Murilo > > > > On Thu, 7 Oct 2021 at 15:33, Boyang Chen <reluctanthero...@gmail.com> > > wrote: > > > > > Hey Murilo, could you explain what you mean by `goka views`? If you are > > > talking about https://github.com/lovoo/goka, they should use consumer > > > groups as well IIUC. > > > > > > Boyang > > > > > > On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares <murilo...@gmail.com> > > > wrote: > > > > > > > Hi. Looking for some insights here. > > > > We use Kafka at a large scale, and have lots of microservices using > > Kafka > > > > for all sorts of things. > > > > Our biggest challenge nowadays is to track which topics are used and > > > which > > > > are not. > > > > I have considered looking at consumer groups to identify which > > > applications > > > > consume from each topic, but then I realized we have lots of > > > microservices > > > > that do not use consumer groups (lots of goka views on top of Kafka > > > > topics). > > > > > > > > Is there any architecture solution that can help track consumers of a > > > Kafka > > > > topic? > > > > Thought that I could use ACL to restrict the creation of Kafka Topics > > and > > > > use a gatekeeper service to centralize and log who creates the > topics, > > > but > > > > still that does not block anyone from consuming from them. > > > > > > > > Any thoughts on that? > > > > Thanks > > > > Murilo > > > > > > > > > >
