Marcel Renders created KAFKA-9746: ------------------------------------- Summary: MM2 produces messages in the wrong cluster Key: KAFKA-9746 URL: https://issues.apache.org/jira/browse/KAFKA-9746 Project: Kafka Issue Type: Bug Components: mirrormaker Affects Versions: 2.4.1, 2.4.0 Reporter: Marcel Renders
When the MirrorSourceConnector (MM2) is used to get messages from a remote Kafka cluster to the Kafka cluster that is connected to Kafka Connect, MM2 is working fine. However, when the connected cluster is being copied to the remote Kafka cluster, the messages are sent to the connected cluster, causing an infinite loop. Note that the topics are in fact created as expected in the remote Kafka cluster and no errors are thrown except flush failures due to the infinite loop. In short: * Connected target -> unconnected source: OK * Connected source -> unconnected target: NOK *Desired situation* The connected Kafka instance should only be used by Kafka Connect to store configs, offsets and status. It should be possible to mirror one Kafka cluster to another while neither are connected to the Kafka Connect instance. Typically the Kafka connectors are either sink or source, but the MM2 is different because the connector settings require both target and source configurations. *Reproducing the issue* This issue occurs with the following configuration, where localhost:9092 is used for Kafka Connect: {code:java} { "name": "MM_TEST", "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "errors.log.enable": "true", "enable.auto.commit": "true", "auto.commit.interval.ms": "20000", "refresh.topics.interval.seconds": "60", "replication.factor": "1", "sync.topic.acls.enabled": "false", "checkpoints.topic.replication.factor": "1", "heartbeats.topic.replication.factor": "1", "offset-syncs.topic.replication.factor": "1", "replication.policy.separator": "", "topics": "'test1'", "tasks.max": "1", "key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "source.cluster.bootstrap.servers": "localhost:9092", "source.cluster.alias": "", "source.cluster.security.protocol": "PLAINTEXT", "source.cluster.group.id": "consumer_mm_test", "target.cluster.bootstrap.servers": "localhost:9093", "target.cluster.alias": "", "target.cluster.security.protocol": "PLAINTEXT" } {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)