Marcel Renders created KAFKA-9746:
-------------------------------------

             Summary: MM2 produces messages in the wrong cluster
                 Key: KAFKA-9746
                 URL: https://issues.apache.org/jira/browse/KAFKA-9746
             Project: Kafka
          Issue Type: Bug
          Components: mirrormaker
    Affects Versions: 2.4.1, 2.4.0
            Reporter: Marcel Renders


When the MirrorSourceConnector (MM2) is used to get messages from a remote 
Kafka cluster to the Kafka cluster that is connected to Kafka Connect, MM2 is 
working fine. However, when the connected cluster is being copied to the remote 
Kafka cluster, the messages are sent to the connected cluster, causing an 
infinite loop.

Note that the topics are in fact created as expected in the remote Kafka 
cluster and no errors are thrown except flush failures due to the infinite loop.

In short:
 * Connected target -> unconnected source: OK
 * Connected source -> unconnected target: NOK

*Desired situation*

The connected Kafka instance should only be used by Kafka Connect to store 
configs, offsets and status. It should be possible to mirror one Kafka cluster 
to another while neither are connected to the Kafka Connect instance.

Typically the Kafka connectors are either sink or source, but the MM2 is 
different because the connector settings require both target and source 
configurations.

*Reproducing the issue*

This issue occurs with the following configuration, where localhost:9092 is 
used for Kafka Connect:
{code:java}
 {
  "name": "MM_TEST",
  "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
  "errors.log.enable": "true",
  "enable.auto.commit": "true",
  "auto.commit.interval.ms": "20000",
  "refresh.topics.interval.seconds": "60",
  "replication.factor": "1",
  "sync.topic.acls.enabled": "false",
  "checkpoints.topic.replication.factor": "1",
  "heartbeats.topic.replication.factor": "1",
  "offset-syncs.topic.replication.factor": "1",
  "replication.policy.separator": "",
  "topics": "'test1'",
  "tasks.max": "1",
  "key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
  "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
  "source.cluster.bootstrap.servers": "localhost:9092",
  "source.cluster.alias": "",
  "source.cluster.security.protocol": "PLAINTEXT",
  "source.cluster.group.id": "consumer_mm_test",
  "target.cluster.bootstrap.servers": "localhost:9093",
  "target.cluster.alias": "",
  "target.cluster.security.protocol": "PLAINTEXT"
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to