Re: Consumer group offset translation in Mirror Maker 2

2023-09-03 Thread Greg Harris
Hey Hemanth, Thanks for your questions! > I'm trying to understand the specific differences between syncing offsets > using RemoteClusterUtils and the inbuilt MM2 consumer offset sync feature. The RemoteClusterUtils reads the checkpoints topic, which is the output of the MirrorCheckpointConnect

Re: Consumer group offset translation in Mirror Maker 2

2023-09-03 Thread Hemanth Savasere
Hi Greg, Thank you for providing the resources to understand offset translation. I'm currently going through them and am gaining a better understanding of the process. Before delving into these resources, I was examining the source code of RemoteClusterUtils.translateOffsets(). I'm trying to unde

Re: Consumer group offset translation in Mirror Maker 2

2023-08-29 Thread Greg Harris
Hey Hemanth! Thank you for asking about Mirror Maker 2! Offset translation is not so simple, so I'll summarize the main functionality and leave some pointers into the code for you to examine yourself. 1. After MirrorSourceTask writes a record, it receives a commitRecord callback [1] with informat

Consumer group offset translation in Mirror Maker 2

2023-08-28 Thread Hemanth Savasere
Hi, We're using Mirror Maker 2 for replicating messages from our primary to secondary cluster, it's one way replication. We're also replicating by the consumer group offsets by adding the below properties sync.group.offsets.enabled=true sync.group.offsets.interval.seconds=5 emit.checkpoints.interv