Hey Daniel, it seems to work fine as well, but it's spamming the error messages multiple times per second into the logs making it hard to put monitoring on the logs and also just steadily growing disk space xD
I've been looking through the code trying to figure out how to disable or change it ... I'm fairly certain that it's hardcoded to use the source cluster: https://github.com/apache/kafka/blob/a9891497314339057bbcfbd43c7715db8d16500e/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java#L380 But it looks like the MirrorCheckpointConnector can be disabled: https://github.com/apache/kafka/blob/a9891497314339057bbcfbd43c7715db8d16500e/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorCheckpointConnector.java#L68 I just can't seem to disable that using strimzi (which we use to deploy kafka in k8s), but maybe that's as simple as doing a small PR to strimzi, so I think I'll go down that route ;) Thanks for pointing me in the right direction at least! Ruben On Tue, Jan 18, 2022 at 8:50 PM Urbán Dániel <urb.dani...@gmail.com> wrote: > Hi Ruben, > > The offset-syncs topic is necessary when the checkpointing feature of > MM2 is being used (a.k.a. consumer group offset replication). > Since the offsets of the original and the replica topic are not > guaranteed to exactly match, there has to be a translation step when > replicating the committed group offsets. The offset-syncs topic is used > for storing these "mappings". > The data replication (MirrorSourceConnector & Task) is responsible for > writing into this topic. > > I don't think there is a way to disable this feature. If you don't use > checkpointing, you can safely ignore the issue. > Alternatively, I remember that there was a feature around configuring > MM2 to put the offset-syncs topic into the target cluster instead of the > source cluster. > > Daniel > > 2022.01.18. 11:01 keltezéssel, Ruben de Vries írta: > > We've been trying to setup a MirrorMaker 2 to mirror data from a cluster > on > > which we only have read access. > > It seems to work, but the logs are hammered with > > TopicAuthorizationException errors in the MirrorCheckpointConnector > thread > > for a mm2-offset-syncs.target-cluster.internal topic. > > > > At this point I think I'm fairly certain it's trying to access this topic > > on the source cluster, which is possible in our case ... > > > > I've been trying to dig through the codebase to figure out what the > purpose > > and importance is of this topic or if it can be disabled, but that's a > bit > > too much of a challenge haha. > > > > So I'm coming here to ask: > > - Could we somehow disable this? > > - Or could we safely ignore this? > > > > Ruben > > > > -- > Az e-mailen az AVG vírusellenőrzést végzett. > http://www.avg.com > >
