SSL Security issue

2021-10-07 Thread Somasuntaram
Hi , I have been trying to enable Kafka security ssl authentication using certificates and encryption. but i am getting errors when i try to create a topic and Kafka status fails whereas zookeeper is running fine. Note :- The screenshot of the error I get when I try to create a topic is attached

Re: Mirror Maker 2: use prefix-less topic names?

2021-10-07 Thread Ryanne Dolan
Jake, in the most recent Kafka 3.0 release you will find IdentityReplicationPolicy, which does what you want. Just be careful that you don't try to replicate the same topics in a loop. Ryanne On Thu, Oct 7, 2021, 1:31 PM Jake Mayward wrote: > Hi, > > I have read a bit on MirrorMaker 2 via > htt

Re: Mirror Maker 2: use prefix-less topic names?

2021-10-07 Thread Men Lim
Nope. Mm2 has that prefix. I read that in 3.0, that prefix is going away. On Thu, Oct 7, 2021 at 11:31 AM Jake Mayward wrote: > Hi, > > I have read a bit on MirrorMaker 2 via > https://kafka.apache.org/documentation/#georeplication-overview, and I am > curious whether I can enable replication wi

Re: Tracking topic consumers

2021-10-07 Thread Murilo Tavares
Oh cool. I had the wrong impression that ACL would not control read/writes, but just topic operations like CREATE and DELETE. Thanks for pointing that out. On Thu, 7 Oct 2021 at 16:09, Boyang Chen wrote: > Thanks, for standalone consumers I do not have any useful monitoring tool > on top of my h

Re: Tracking topic consumers

2021-10-07 Thread Boyang Chen
Thanks, for standalone consumers I do not have any useful monitoring tool on top of my head, maybe this KIP looks promising to you, but it's still WIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability Also regarding ACL setup, I think that's also a good

Re: Tracking topic consumers

2021-10-07 Thread Murilo Tavares
Hi Boyang Thanks for your response. Yes, I'm talking about that framework. goka has a few different things. What they call `views` are similar to Global KTables in KafkaStreams. It's a full snapshot of the Kafka topic, held in memory. So every time you spin up your service, the goka view will read

Re: Tracking topic consumers

2021-10-07 Thread Boyang Chen
Hey Murilo, could you explain what you mean by `goka views`? If you are talking about https://github.com/lovoo/goka, they should use consumer groups as well IIUC. Boyang On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares wrote: > Hi. Looking for some insights here. > We use Kafka at a large scale,

Tracking topic consumers

2021-10-07 Thread Murilo Tavares
Hi. Looking for some insights here. We use Kafka at a large scale, and have lots of microservices using Kafka for all sorts of things. Our biggest challenge nowadays is to track which topics are used and which are not. I have considered looking at consumer groups to identify which applications cons

Mirror Maker 2: use prefix-less topic names?

2021-10-07 Thread Jake Mayward
Hi, I have read a bit on MirrorMaker 2 via https://kafka.apache.org/documentation/#georeplication-overview, and I am curious whether I can enable replication with MirrorMaker 2 without having to prefix the topic with the cluster name. Reasoning behind this is that I would like the client to al