Mirrormaker 2.0 and compacted topics

Hello, We're currently trying to evaluate Mirrormaker 2.0 for future inter-cluster replication, replacing our bespoke replicator. I understand that Mirrormaker 2.0 documentation is only slated to be released in Kafka 2.5.0, but I was hoping that someone will know whether Mirrormaker 2.0 can be

Re: Connect - Membership Protocol

Are the four brokers across two clusters (B1+B2) / (B3+B4), or one cluster? If one cluster, are using the same config/offset/status topics for each Connect cluster? Because that definitely won't work. In case it's useful: https://rmoff.net/2019/11/22/common-mistakes-made-when-configuring-multiple-

Re: Mirrormaker 2.0 and compacted topics

Hi Pirow, as far as I understand MirrorMaker 2.0 will not treat compacted topics any different from uncompacted topics. What that means for your scenario is that your replication may miss some messages in the case of a long unavailability, if those messages were compacted in the meantime. However

RE: Mirrormaker 2.0 and compacted topics

Hi Sönke, OK, thanks, so compacted topics is supported, but an exact replica (i.e. Kafka records at the same offsets) of the original topic is not possible as there is nothing to replicate. Is my understanding correct? Thanks Pirow Engelbrecht | Senior Design Engineer Tel +27 12 678 9740 (ext.

Re: Mirrormaker 2.0 and compacted topics

Hi Pirow, records at the same offset as in the original topic is not possible for non compacted topics either, that is one of the major constraints of MirrorMaker. Compaction doesn't change offsets of messages, so in theory should behave the same as a non compacted topic, unless of course you mis

RE: Mirrormaker 2.0 and compacted topics

Hi Sönke, Thank you for the clarification – much appreciated. Kind regards Pirow Engelbrecht | Senior Design Engineer Tel +27 12 678 9740 (ext. 9879) | Cell +27 63 148 3376 76 Regency Drive | Irene | Centurion | 0157 [cid:image001.jpg@01D5FD28.3140B0E0]

Re: MirrorMaker2 not mirroring for 5 minutes when adding a topic

Peter, can you share any log lines like "x took y ms" or "x took too long (y ms)" or "timed out running task x"? Ryanne On Tue, Mar 17, 2020 at 10:45 AM Péter Sinóros-Szabó wrote: > Hey, > > Running a MM2 cluster to mirror from A->B clusters I noticed that when I > add a new topic to A cluster,

Kafka Streams - partition assignment for the input topic

I have a question about partition assignment for a kafka streams app. As I understand it the more complex your topology is the greater the number of internal topics kafka streams will create. In my case the app has 8 graphs in the topology. There are 6 partitions for each graph (this matches the nu

I'm trying to connect to a kafka broker running in AWS EKS (from outside the EKS cluster).

Problem: I'm hitting an error: "no such host" for " kafka-0.cluster.local:19092". Has anyone done this before? Any help would be appreciated. Thanks! - Dan My long-term goal is to get an AWS Lambda to send events to a Kafka running in AWS EKS. I used the following instructions

Regarding segment size config

Hi community, I understand that there are two configs regarding segment file size, log.segment.bytes for broker and segment.bytes for topic. The default values are both 1G and they are required to be an integer so they cannot be larger than 2G. My question is, assuming I am not making any mistakes

RE: I'm trying to connect to a kafka broker running in AWS EKS (from outside the EKS cluster).

Hi Dan, For Kafka: * Make sure that your listeners and advertised listener configuration is set up correctly for external connectivity, incl. your Kafka protocols For Kubernetes: * How are you exposing the Kafka port? Your port number seems low for Kubernetes, typically external ports

Kafka JMX monitoring

Hi, I want to know what the best practice to collect Kafka JMX metrics is. I haven't found a decent way to collect and parse JMX in Java (because it is too much) and I learn that there are tools like tools like jmxtrans to do this. I wonder if there is more. Thanks. Regards.