Re: MirrorMaker 2 without write access on source

2022-01-26 Thread Mickael Maison
Hi, This is possible since Kafka 3.0. You can use the "offset-syncs.topic.location" configuration on both the source and checkpoint connectors to move the offset-syncs topic to the target cluster. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-716%3A+Allow+configuring+the+location+of+

Re: MirrorMaker 2 without write access on source

2022-01-26 Thread Ruben de Vries
Hey Daniel, it seems to work fine as well, but it's spamming the error messages multiple times per second into the logs making it hard to put monitoring on the logs and also just steadily growing disk space xD I've been looking through the code trying to figure out how to disable or change it ...

Re: MirrorMaker 2 without write access on source

2022-01-18 Thread Urbán Dániel
Hi Ruben, The offset-syncs topic is necessary when the checkpointing feature of MM2 is being used (a.k.a. consumer group offset replication). Since the offsets of the original and the replica topic are not guaranteed to exactly match, there has to be a translation step when replicating the com

MirrorMaker 2 without write access on source

2022-01-18 Thread Ruben de Vries
We've been trying to setup a MirrorMaker 2 to mirror data from a cluster on which we only have read access. It seems to work, but the logs are hammered with TopicAuthorizationException errors in the MirrorCheckpointConnector thread for a mm2-offset-syncs.target-cluster.internal topic. At this poin