Hi Boyang Thanks for your response. Yes, I'm talking about that framework. goka has a few different things. What they call `views` are similar to Global KTables in KafkaStreams. It's a full snapshot of the Kafka topic, held in memory. So every time you spin up your service, the goka view will read the topic from beginning, thus no need to keep track of consumer offset or consumer groups. Thanks Murilo
On Thu, 7 Oct 2021 at 15:33, Boyang Chen <reluctanthero...@gmail.com> wrote: > Hey Murilo, could you explain what you mean by `goka views`? If you are > talking about https://github.com/lovoo/goka, they should use consumer > groups as well IIUC. > > Boyang > > On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares <murilo...@gmail.com> > wrote: > > > Hi. Looking for some insights here. > > We use Kafka at a large scale, and have lots of microservices using Kafka > > for all sorts of things. > > Our biggest challenge nowadays is to track which topics are used and > which > > are not. > > I have considered looking at consumer groups to identify which > applications > > consume from each topic, but then I realized we have lots of > microservices > > that do not use consumer groups (lots of goka views on top of Kafka > > topics). > > > > Is there any architecture solution that can help track consumers of a > Kafka > > topic? > > Thought that I could use ACL to restrict the creation of Kafka Topics and > > use a gatekeeper service to centralize and log who creates the topics, > but > > still that does not block anyone from consuming from them. > > > > Any thoughts on that? > > Thanks > > Murilo > > >
