Hi Mickael, thanks for granting the rights. Right now this is exactly what I did for my local copy: I've deactivated the creation of the topics and the overall offset sync process once "emit.checkpoints.enabled" was set to false.
Do you think of a separate configuration for running the offset sync? Like "emit.offset-syncs.enabled"? And when this one is enabled it also creates the offset sync topic and when the "emit.checkpoints.enabled" is set to true it creates the "checkpoints" topic? Of course one need to make sure that setting "emit.checkpoints.enabled" to "true" and setting "emit.offset-syncs.enabled" to "false" would be illegal. Or do you have something completely different in mind? Kind regards Georg Friedrich -----Original Message----- From: Mickael Maison <mickael.mai...@gmail.com> Sent: Thursday, March 18, 2021 5:47 PM To: dev <dev@kafka.apache.org> Subject: Re: MirrorMaker 2.0 - Offset Sync - Questions/Improvements Hi Georg, I've granted you contributor access, you should now be able to assign JIRAs to yourself. The offset-syncs topic is created and populated by the Source connector. I'm not sure if setting "emit.checkpoints.enabled" to false on that connector is the most intuitive. How would you disable that function? Thanks On Thu, Mar 18, 2021 at 3:42 PM Georg Friedrich <georg.friedr...@webfleet.com> wrote: > > Ok, can anybody grant me access to the Kafka JIRA so I can work on the ticket > I'm going to create. > My username is "GFriedrich". > > Thanks in advance. > > Kind regards > Georg Friedrich > > -----Original Message----- > From: Ryanne Dolan <ryannedo...@gmail.com> > Sent: Thursday, March 18, 2021 3:46 PM > To: dev <dev@kafka.apache.org> > Subject: Re: MirrorMaker 2.0 - Offset Sync - Questions/Improvements > > I am not certain, but I'd guess a KIP is not required. I think this would be > considered a bug fix. > > Ryanne > > On Wed, Mar 17, 2021, 5:34 PM Georg Friedrich > <georg.friedr...@webfleet.com> > wrote: > > > Hi Ryanne, > > > > thank you for your response. > > > > 1) Yes, right - I've missed the condition that is part of the > > PartitionState class. Thanks for pointing me. :) > > > > 2) Ok, how should I go on there? Shall I create a ticket or a KIP or > > even both? From my point of view this is not a major change, but for > > people relying on the fact that those topics are always created this > > may look different. > > > > Kind regards > > Georg Friedrich > > > > -----Original Message----- > > From: Ryanne Dolan <ryannedo...@gmail.com> > > Sent: Wednesday, March 17, 2021 4:57 AM > > To: dev <dev@kafka.apache.org> > > Subject: Re: MirrorMaker 2.0 - Offset Sync - Questions/Improvements > > > > Georg, sorry for the delay, but hopefully I can still help. > > > > 1) I think the detail you're missing is that the offset syncs are > > very sparse. Normally, you only get a new sync when the Connector > > first starts running. You are right that it is possible for a > > consumer to lag behind the most recent offset sync, but that will be a > > rare, transient condition, e.g. > > when the Connector first starts running. > > > > 2) I think you are right -- disabling checkpoints probably should > > also prevent those topics from being created. I'd support that change. > > > > Ryanne > > > > On Fri, Feb 26, 2021, 4:24 PM Georg Friedrich < > > georg.friedr...@webfleet.com> > > wrote: > > > > > Hi, > > > > > > recently I've started to look deeper into the code of MirrorMaker > > > 2.0 and was faced with some confusing details. Maybe you can point > > > me into a right direction here. > > > > > > > > > * The line at > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > > gi > > > th > > > ub.com%2Fapache%2Fkafka%2Fblob%2F02226fa090513882b9229ac834fd493d7 > > > 1a > > > e6 > > > d96%2Fconnect%2Fmirror%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkafka% > > > 2F > > > co > > > nnect%2Fmirror%2FOffsetSyncStore.java%23L52&data=04%7C01%7Cgeo > > > rg > > > .f > > > riedrich%40webfleet.com%7Cf53651fc33834e4a793d08d8e8f8c563%7Ce648a > > > 63 > > > 41 > > > 151497c97970f975bddecc0%7C0%7C0%7C637515502445685286%7CUnknown%7CT > > > WF > > > pb > > > GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI > > > 6M > > > n0 > > > %3D%7C1000&sdata=D3%2BdZuT3d7wvqLN%2F3q5zIi07iwI7ZO1doxVq2NSHj > > > WU > > > %3 > > > D&reserved=0 checks whether the offsets that get translated > > > are smaller than the last offset sync. > > > If this is the case, no translation happens. But I'm confused here: > > > Isn't this a potential issue? What if some consumers are slow in > > > regards to processing messages from Kafka and fall back behand the > > > offset sync process of the MirrorMaker. > > > In this case the MirrorMaker would stop to translate any offsets. > > > Do I miss something here or is this really broken? > > > * I'm wondering: One is able to deactivate emitting checkpoints to > > the > > > target cluster. But when this happens, the offset sync topic is > > > still written to the source cluster. Why is that? As far as I can > > > see the only consumer of the offset sync topic is the checkpoint > > > connector. So one could also deactivate the whole offset sync > > > production entirely when disabling emitting checkpoints. Or is > > > there again something that I miss? If not, is this worth a KIP? > > > > > > Thanks in advance for your answers and help. > > > > > > Kind regards > > > Georg Friedrich > > > > >
