[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929359#comment-16929359
]
Ryanne Dolan commented on KAFKA-7500:
-------------------------------------
[~chridtian.hagel] thanks for giving it a spin. The "failed to flush" errors
are probably due to WorkerSourceTask being unable to send the 5942 messages
within the default flush timeout, which I believe is 5 seconds. There are
various reasons this might be the case:
- tasks.max could be 1 (the default), which means a single Producer is sending
records across the entire Herder. Try increasing this considerably. This can be
as high as the total number of partitions being replicated, at the cost of more
overhead per partition, obviously. If you configure this too high, MM2 just
uses one task per partition.
- The producer lag may be high, which is detrimental to throughput. Make sure
the MM2 driver is running close to the target cluster to minimize this latency.
If you are replicating between multiple DCs, consider running a few MM2 nodes
in each DC, with `--clusters` argument to hint which clusters are nearby. That
way, drivers will consume from other DCs but only produce locally.
- You may need to use more MM2 nodes.
- You may need to increase the 5 second flush timeout.
Re: duplicated messages, you are correct that MM2 will send dupes if containers
are bounced like that. Generally, this is okay -- occasional dupes are a fact
of life in most Kafka pipelines. That said, I am working on a PoC and KIP for
exactly-once replication with MM2, which will eliminate these dupes.
> MirrorMaker 2.0 (KIP-382)
> -------------------------
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
> Affects Versions: 2.4.0
> Reporter: Ryanne Dolan
> Assignee: Manikumar
> Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
