Georg, sorry for the delay, but hopefully I can still help. 1) I think the detail you're missing is that the offset syncs are very sparse. Normally, you only get a new sync when the Connector first starts running. You are right that it is possible for a consumer to lag behind the most recent offset sync, but that will be a rare, transient condition, e.g. when the Connector first starts running.
2) I think you are right -- disabling checkpoints probably should also prevent those topics from being created. I'd support that change. Ryanne On Fri, Feb 26, 2021, 4:24 PM Georg Friedrich <georg.friedr...@webfleet.com> wrote: > Hi, > > recently I've started to look deeper into the code of MirrorMaker 2.0 and > was faced with some confusing details. Maybe you can point me into a right > direction here. > > > * The line at > https://github.com/apache/kafka/blob/02226fa090513882b9229ac834fd493d71ae6d96/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/OffsetSyncStore.java#L52 > checks whether the offsets that get translated are smaller than the last > offset sync. > If this is the case, no translation happens. But I'm confused here: Isn't > this a potential issue? What if some consumers are slow in regards to > processing messages from Kafka and fall back behand the offset sync process > of the MirrorMaker. > In this case the MirrorMaker would stop to translate any offsets. Do I > miss something here or is this really broken? > * I'm wondering: One is able to deactivate emitting checkpoints to the > target cluster. But when this happens, the offset sync topic is still > written to the source cluster. Why is that? As far as I can see the only > consumer of the offset sync topic is the checkpoint connector. So one could > also deactivate the whole offset sync production entirely when disabling > emitting checkpoints. Or is there again something that I miss? If not, is > this worth a KIP? > > Thanks in advance for your answers and help. > > Kind regards > Georg Friedrich >
