You may be running into the issue described here:

https://issues.apache.org/jira/browse/KAFKA-16364

If that's the case, I believe you have two options:

1. Reduce emit.checkpoints.interval.seconds and 
sync.group.offsets.interval.seconds to reduce the offset lag between source and 
target clusters while consumers are active on the source cluster
2. Manually update offsets on the target cluster once consumers are stopped on 
the source cluster (e.g. using kafka-consumer-groups.sh)

A bit more background on this question based on my understanding:

https://github.com/aws-samples/mirrormaker2-msk-migration?tab=readme-ov-file#how-can-i-minimize-the-number-of-duplicate-messages-i-process-in-my-consumers


On 10/4/24, 8:13 AM, "Massimiliano Rotondo" <scuta1...@gmail.com 
<mailto:scuta1...@gmail.com>> wrote:


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.






Hello,


I'm experiencing a side bug with usage of Kafka MirrorMaker2, all is
described in the below post:


https://stackoverflow.com/questions/79051730/kafka-mirrormaker2-stop-offset-sync-when-consumer-group-idle-on-source-cluster
 
<https://stackoverflow.com/questions/79051730/kafka-mirrormaker2-stop-offset-sync-when-consumer-group-idle-on-source-cluster>


The same "bug" has been reproduced using updated Kafka Cluster 3.7.1 on
both source/destination (so it seems not depending on the old version of
the source cluster).


Could you please support with this?


Best Regards,


Massimiliano Rotondo



Reply via email to