RE: RE: RE: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0

2020-12-20 Thread Amit.SRIVASTAV
Hi Ning and all, We got a crude way to solve this issue. Below are the high level steps: Read the message from Replicator's internal topic for storing offsets. [connect-offsets] This topic stores the offsets for all topics which is getting replicated in key:value pair . For e.g. Key : ["replica

RE: RE: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0

2020-12-07 Thread Amit.SRIVASTAV
Hi Ning, It did not worked. Here are the logs from the replicator and mirror maker 2 respectively: Replicator: [2020-12-08 05:11:06,611] INFO [Consumer clientId=onprem-aws-replicator-0, groupId=onprem-aws-replicator] Seeking to offset 83 for partition ONPREM.AWS.REPLICA.TOPIC.P3R3-0 (org.a

RE: Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0

2020-12-06 Thread Amit.SRIVASTAV
Hi Ning, Thank you for the response. I probably tried to change every possible consumer group id in all ways MM2 can run. But I went through the code and it seems consumer config is using assign method which does not requires group id to be assigned and also could not find anywhere that proper

Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0

2020-11-25 Thread Amit.SRIVASTAV
Hi All, We are currently trying to migrate Confluent replicator to Apache Open Source Mirror Maker v2.0. We are facing an issue where the messages which are already replicated by replicator is getting replicated again when the mirror maker is started on the same topic. This should not happen as