Re: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0

[Ning Zhang]

Hi Amit, I guess you may need to override the actual consumer group config 
(probably not consumer.group.id) that is used in Kafka Connect 

On 2020/11/26 06:47:11, <amit.srivas...@cognizant.com> wrote: 
> Hi All,
> 
> We are currently trying to migrate Confluent replicator to Apache Open Source 
> Mirror Maker v2.0. We are facing an issue where the messages which are 
> already replicated by replicator is getting replicated again when the mirror 
> maker is started on the same topic. This should not happen as messages are 
> getting duplicated at the target cluster. Here are more details:
> 
> 
> 1.       RCA: replicator assign a consumer group for replicating messages. 
> This consumer group maintains the offset of the source topic. But we are not 
> able to assign same consumer group to the Consumer config in mirror maker 2.
> 
> 2.       Mirror Maker 1.0 : working as same consumer group can be assigned in 
> consumer.properties file and the messages are picked right after where 
> replicator was stopped.
> 
> 3.       Tried running and configuring source.cluster.consumer.group.id in 
> mirror maker 2.0 in all available options (in cluster mode, in 
> connect-standalone and connect-distributed mode) but mirror maker 2.0 is 
> assigning consumer group id as null while replicating messages.
> 
> 
> Any pointers if anyone has done same and tried to maintain the same offset 
> with mirror maker 2.0.
> 
> Thanks and regards,
> Amit
> This e-mail and any files transmitted with it are for the sole use of the 
> intended recipient(s) and may contain confidential and privileged 
> information. If you are not the intended recipient(s), please reply to the 
> sender and destroy all copies of the original message. Any unauthorized 
> review, use, disclosure, dissemination, forwarding, printing or copying of 
> this email, and/or any action taken in reliance on the contents of this 
> e-mail is strictly prohibited and may be unlawful. Where permitted by 
> applicable law, this e-mail and other e-mail communications sent to and from 
> Cognizant e-mail addresses may be monitored.
>