Hi, This is possible since Kafka 3.0. You can use the "offset-syncs.topic.location" configuration on both the source and checkpoint connectors to move the offset-syncs topic to the target cluster.
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-716%3A+Allow+configuring+the+location+of+the+offset-syncs+topic+with+MirrorMaker2 Thanks, Mickael On Wed, Jan 26, 2022 at 9:39 AM Ruben de Vries <ru...@provablelabs.com> wrote: > > Hey Daniel, > > it seems to work fine as well, but it's spamming the error messages > multiple times per second into the logs making it hard to put monitoring on > the logs and also just steadily growing disk space xD > > I've been looking through the code trying to figure out how to disable or > change it ... > > I'm fairly certain that it's hardcoded to use the source cluster: > https://github.com/apache/kafka/blob/a9891497314339057bbcfbd43c7715db8d16500e/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L380 > > But it looks like the MirrorCheckpointConnector can be disabled: > https://github.com/apache/kafka/blob/a9891497314339057bbcfbd43c7715db8d16500e/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointConnector.java#L68 > I just can't seem to disable that using strimzi (which we use to deploy > kafka in k8s), but maybe that's as simple as doing a small PR to strimzi, > so I think I'll go down that route ;) > > Thanks for pointing me in the right direction at least! > > Ruben > > > > > > > > On Tue, Jan 18, 2022 at 8:50 PM Urbán Dániel <urb.dani...@gmail.com> wrote: > > > Hi Ruben, > > > > The offset-syncs topic is necessary when the checkpointing feature of > > MM2 is being used (a.k.a. consumer group offset replication). > > Since the offsets of the original and the replica topic are not > > guaranteed to exactly match, there has to be a translation step when > > replicating the committed group offsets. The offset-syncs topic is used > > for storing these "mappings". > > The data replication (MirrorSourceConnector & Task) is responsible for > > writing into this topic. > > > > I don't think there is a way to disable this feature. If you don't use > > checkpointing, you can safely ignore the issue. > > Alternatively, I remember that there was a feature around configuring > > MM2 to put the offset-syncs topic into the target cluster instead of the > > source cluster. > > > > Daniel > > > > 2022.01.18. 11:01 keltezéssel, Ruben de Vries írta: > > > We've been trying to setup a MirrorMaker 2 to mirror data from a cluster > > on > > > which we only have read access. > > > It seems to work, but the logs are hammered with > > > TopicAuthorizationException errors in the MirrorCheckpointConnector > > thread > > > for a mm2-offset-syncs.target-cluster.internal topic. > > > > > > At this point I think I'm fairly certain it's trying to access this topic > > > on the source cluster, which is possible in our case ... > > > > > > I've been trying to dig through the codebase to figure out what the > > purpose > > > and importance is of this topic or if it can be disabled, but that's a > > bit > > > too much of a challenge haha. > > > > > > So I'm coming here to ask: > > > - Could we somehow disable this? > > > - Or could we safely ignore this? > > > > > > Ruben > > > > > > > -- > > Az e-mailen az AVG vírusellenőrzést végzett. > > http://www.avg.com > > > >
