Logs with the Disconnect Exception (when i set the loglevel to - INFO) :
```
[SourceTaskOffsetCommitter-1]
2023-02-23 22:54:13,546 INFO [Consumer clientId=consumer-null-4,
groupId=null] Error sending fetch request (sessionId=1509420532, epoch=492)
to node 2: (org.apache.kafka.clients.FetchSessionHandler)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
org.apache.kafka.common.errors.DisconnectException
2023-02-23 22:54:13,711 WARN [Consumer clientId=consumer-null-4,
groupId=null] Connection to node 2
(nossl-w-kafka-2.nossl-w-kafka-brokers.kafka.svc/10.98.129.75:9092) could
not be established. Broker may not be available.
(org.apache.kafka.clients.NetworkClient)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
2023-02-23 22:54:13,711 INFO [Consumer clientId=consumer-null-4,
groupId=null] Error sending fetch request (sessionId=1509420532,
epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
org.apache.kafka.common.errors.DisconnectException
2023-02-23 22:54:24,744 INFO
WorkerSourceTask{id=my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0}
flushing 0 outstanding messages for offset commit
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[SourceTaskOffsetCommitter-1]
2023-02-23 22:54:37,029 INFO [Consumer clientId=consumer-null-4,
groupId=null] Error sending fetch request (sessionId=1509420532,
epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
org.apache.kafka.common.errors.DisconnectException
2023-02-23 22:55:06,431 INFO
WorkerSourceTask{id=my-source-cluster-west->my-target-cluster-east.MirrorHeartbeatConnector-0}
flushing 0 outstanding messages for offset commit
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[SourceTaskOffsetCommitter-1]
2023-02-23 22:55:07,449 INFO [Consumer clientId=consumer-null-4,
groupId=null] Error sending fetch request (sessionId=1509420532,
epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
org.apache.kafka.common.errors.TimeoutException: Failed to send request
after 30000 ms.
2023-02-23 22:55:10,029 INFO [Consumer clientId=consumer-null-4,
groupId=null] Error sending fetch request (sessionId=1509420532,
epoch=INITIAL) to node 2: (org.apache.kafka.clients.FetchSessionHandler)
[task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
org.apache.kafka.common.errors.DisconnectException
2023-02-23 22:55:24,745 INFO
WorkerSourceTask{id=my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0}
flushing 0 outstanding messages for offset commit
(org.apache.kafka.connect.runtime.WorkerSourceTask)
[SourceTaskOffsetCommitter-1]
```
On Thu, Feb 23, 2023 at 2:12 PM karan alang <karan.al...@gmail.com> wrote:
> Hello All,
>
> Anyone has installed KafkaMirrorMaker2 on GKE ?
> Need some help to debug/resolve issues that I've been having.
>
> I've 2 clusters on GKE with strimzi kafka installed.
> cluster1(nossl-e) on use-east1, cluster2(nossl-e) on us-west1.
>
> Both the clusters are Auto-pilot cluster (mentioning this, thought it
> might not be relevant for the issues being faced)
>
> source cluster -> nossl-w
> target cluster -> nossl-e
>
> Currently(to get MM2 working), SSL & authorization are not enabled.
>
> KafkaMirrorMaker2 is installed on source cluster i.e. nossl-w
>
> Here is the yaml :
> ```
> apiVersion: kafka.strimzi.io/v1beta2
> kind: KafkaMirrorMaker2
> metadata:
> name: my-mirror-maker-2
> spec:
> version: 3.0.0
> replicas: 1
> connectCluster: "my-target-cluster-east"
> clusters:
> - alias: "my-source-cluster-west"
> bootstrapServers: nossl-w-kafka-bootstrap:9092
> - alias: "my-target-cluster-east"
> bootstrapServers: xx.xx.xx.xx:9094
> config:
> # -1 means it will use the default replication factor configured in
> the broker
> config.storage.replication.factor: -1
> offset.storage.replication.factor: -1
> status.storage.replication.factor: -1
> mirrors:
> - sourceCluster: "my-source-cluster-west"
> targetCluster: "my-target-cluster-east"
> sourceConnector:
> config:
> replication.factor: 1
> offset-syncs.topic.replication.factor: 1
> sync.topic.acls.enabled: "false"
> heartbeatConnector:
> config:
> heartbeats.topic.replication.factor: 1
> checkpointConnector:
> config:
> checkpoints.topic.replication.factor: 1
> topicsPattern: ".*"
> groupsPattern: ".*"
>
> ```
>
> While the pods & the svc come up, MirrorMaker is not transferring data
> from source to target topics. (mmtest - in my case, that is the only topic
> created on both the clusters)
>
> Here are the logs from the mirrormaker pod :
>
>
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Requesting metadata update for partition
> mirrormaker2-cluster-offsets-7 due to error LEADER_NOT_AVAILABLE
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition
> mirrormaker2-cluster-offsets-21 from 226 to epoch 226 from new metadata
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition
> mirrormaker2-cluster-status-0 from 225 to epoch 225 from new metadata
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Requesting metadata update for partition
> mirrormaker2-cluster-status-0 due to error LEADER_NOT_AVAILABLE
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition
> mirrormaker2-cluster-status-1 from 226 to epoch 226 from new metadata
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition
> mirrormaker2-cluster-status-4 from 226 to epoch 226 from new metadata
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition
> mirrormaker2-cluster-status-2 from 226 to epoch 226 from new metadata
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition
> mirrormaker2-cluster-status-3 from 225 to epoch 225 from new metadata
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Requesting metadata update for partition
> mirrormaker2-cluster-status-3 due to error LEADER_NOT_AVAILABLE
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition mmtest-0 from 460 to
> epoch 460 from new metadata (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition mmtest-2 from 416 to
> epoch 416 from new metadata (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition mmtest-1 from 433 to
> epoch 433 from new metadata (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updating last seen epoch for partition
> mirrormaker2-cluster-configs-0 from 226 to epoch 226 from new metadata
> (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,632 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Updated cluster metadata updateVersion 1597 to
> MetadataCache{clusterId='yoD3YJaKTz6dvaVT2MfiOw',
> nodes={1=nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1 rack:
> null), 2=nossl-w-kafka-2.nossl-w-kafka-brokers.kafka.svc:9092 (id: 2 rack:
> null)}, partitions=[PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-offsets-0, leader=Optional[1],
> leaderEpoch=Optional[226], replicas=1, isr=1, offlineReplicas=),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-status-3, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-2,
> leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
> offlineReplicas=), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-status-1, leader=Optional[2],
> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-offsets-4, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mmtest-0, leader=Optional[2],
> leaderEpoch=Optional[460], replicas=2,1,0, isr=1,2, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-6,
> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
> offlineReplicas=), PartitionMetadata(error=NONE, partition=mmtest-2,
> leader=Optional[2], leaderEpoch=Optional[416], replicas=0,2,1, isr=1,2,
> offlineReplicas=0), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-offsets-8, leader=Optional[2],
> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-offsets-10, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-12,
> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
> offlineReplicas=), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-offsets-14, leader=Optional[2],
> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-15,
> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
> offlineReplicas=), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-offsets-17, leader=Optional[2],
> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-offsets-19, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-21,
> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
> offlineReplicas=), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-offsets-23, leader=Optional[2],
> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-offsets-1, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-status-4,
> leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
> offlineReplicas=), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-offsets-3, leader=Optional[1],
> leaderEpoch=Optional[226], replicas=1, isr=1, offlineReplicas=),
> PartitionMetadata(error=NONE, partition=mmtest-1, leader=Optional[1],
> leaderEpoch=Optional[433], replicas=1,0,2, isr=1,2, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-status-2,
> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
> offlineReplicas=), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-offsets-5, leader=Optional[2],
> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-status-0, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-offsets-7, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-9,
> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
> offlineReplicas=), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-offsets-11, leader=Optional[2],
> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-offsets-13, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-offsets-16, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-18,
> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
> offlineReplicas=), PartitionMetadata(error=NONE,
> partition=mirrormaker2-cluster-configs-0, leader=Optional[2],
> leaderEpoch=Optional[226], replicas=2, isr=2, offlineReplicas=),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-20,
> leader=Optional[2], leaderEpoch=Optional[226], replicas=2, isr=2,
> offlineReplicas=), PartitionMetadata(error=LEADER_NOT_AVAILABLE,
> partition=mirrormaker2-cluster-offsets-22, leader=Optional.empty,
> leaderEpoch=Optional[225], replicas=0, isr=0, offlineReplicas=0),
> PartitionMetadata(error=NONE, partition=mirrormaker2-cluster-offsets-24,
> leader=Optional[1], leaderEpoch=Optional[226], replicas=1, isr=1,
> offlineReplicas=)],
> controller=nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1
> rack: null)} (org.apache.kafka.clients.Metadata)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
>
> 2023-02-23 22:05:54,666 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Received FETCH response from node 1 for request with header
> RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=consumer-mirrormaker2-cluster-1, correlationId=12072):
> FetchResponseData(throttleTimeMs=0, errorCode=0, sessionId=1476981493,
> responses=[]) (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work
> Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,666 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Node 1 sent an incremental fetch response with throttleTimeMs = 0 for
> session 1476981493 with 0 response partition(s), 8 implied partition(s)
> (org.apache.kafka.clients.FetchSessionHandler) [KafkaBasedLog Work Thread -
> mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-2 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-8 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-17 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-23 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-5 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-11 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-20 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-14 at position FetchPosition{offset=2,
> offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
> 34.75.2.112:9094 (id: 1 rack: null)], epoch=0}} to node 34.75.2.112:9094
> (id: 1 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Built incremental fetch (sessionId=1476981493, epoch=4013) for node 1.
> Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of
> 8 partition(s) (org.apache.kafka.clients.FetchSessionHandler)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(),
> implied=(mirrormaker2-cluster-offsets-2, mirrormaker2-cluster-offsets-8,
> mirrormaker2-cluster-offsets-14, mirrormaker2-cluster-offsets-17,
> mirrormaker2-cluster-offsets-23, mirrormaker2-cluster-offsets-5,
> mirrormaker2-cluster-offsets-11, mirrormaker2-cluster-offsets-20)) to
> broker 34.75.2.112:9094 (id: 1 rack: null)
> (org.apache.kafka.clients.consumer.internals.Fetcher) [KafkaBasedLog Work
> Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,667 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Sending FETCH request with header RequestHeader(apiKey=FETCH,
> apiVersion=12, clientId=consumer-mirrormaker2-cluster-1,
> correlationId=12075) and timeout 30000 to node 1:
> FetchRequestData(clusterId=null, replicaId=-1, maxWaitMs=500, minBytes=1,
> maxBytes=52428800, isolationLevel=0, sessionId=1476981493,
> sessionEpoch=4013, topics=[], forgottenTopicsData=[], rackId='')
> (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread -
> mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Received FETCH response from node 0 for request with header
> RequestHeader(apiKey=FETCH, apiVersion=12,
> clientId=consumer-mirrormaker2-cluster-1, correlationId=12073):
> FetchResponseData(throttleTimeMs=0, errorCode=0, sessionId=1395933211,
> responses=[]) (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work
> Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Node 0 sent an incremental fetch response with throttleTimeMs = 0 for
> session 1395933211 with 0 response partition(s), 9 implied partition(s)
> (org.apache.kafka.clients.FetchSessionHandler) [KafkaBasedLog Work Thread -
> mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-0 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-12 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-15 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-21 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-18 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-24 at position FetchPosition{offset=0,
> offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-6 at position FetchPosition{offset=1,
> offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-3 at position FetchPosition{offset=2,
> offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Added READ_UNCOMMITTED fetch request for partition
> mirrormaker2-cluster-offsets-9 at position FetchPosition{offset=1,
> offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[
> 35.231.229.1:9094 (id: 0 rack: null)], epoch=0}} to node 35.231.229.1:9094
> (id: 0 rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Built incremental fetch (sessionId=1395933211, epoch=4010) for node 0.
> Added 0 partition(s), altered 0 partition(s), removed 0 partition(s) out of
> 9 partition(s) (org.apache.kafka.clients.FetchSessionHandler)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(), toForget=(),
> implied=(mirrormaker2-cluster-offsets-0, mirrormaker2-cluster-offsets-6,
> mirrormaker2-cluster-offsets-12, mirrormaker2-cluster-offsets-15,
> mirrormaker2-cluster-offsets-21, mirrormaker2-cluster-offsets-3,
> mirrormaker2-cluster-offsets-9, mirrormaker2-cluster-offsets-18,
> mirrormaker2-cluster-offsets-24)) to broker 35.231.229.1:9094 (id: 0
> rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher)
> [KafkaBasedLog Work Thread - mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,692 DEBUG [Consumer
> clientId=consumer-mirrormaker2-cluster-1, groupId=mirrormaker2-cluster]
> Sending FETCH request with header RequestHeader(apiKey=FETCH,
> apiVersion=12, clientId=consumer-mirrormaker2-cluster-1,
> correlationId=12076) and timeout 30000 to node 0:
> FetchRequestData(clusterId=null, replicaId=-1, maxWaitMs=500, minBytes=1,
> maxBytes=52428800, isolationLevel=0, sessionId=1395933211,
> sessionEpoch=4010, topics=[], forgottenTopicsData=[], rackId='')
> (org.apache.kafka.clients.NetworkClient) [KafkaBasedLog Work Thread -
> mirrormaker2-cluster-offsets]
>
> 2023-02-23 22:05:54,699 DEBUG [Worker clientId=connect-1,
> groupId=mirrormaker2-cluster] Received HEARTBEAT response from node
> 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT,
> apiVersion=4, clientId=connect-1, correlationId=780):
> HeartbeatResponseData(throttleTimeMs=0, errorCode=0)
> (org.apache.kafka.clients.NetworkClient) [DistributedHerder-connect-1-1]
>
> 2023-02-23 22:05:54,699 DEBUG [Worker clientId=connect-1,
> groupId=mirrormaker2-cluster] Received successful Heartbeat response
> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator)
> [DistributedHerder-connect-1-1]
>
> 2023-02-23 22:05:54,732 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Sending metadata request
> MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
> name='mirrormaker2-cluster-offsets'),
> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
> name='mirrormaker2-cluster-status'),
> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='mmtest'),
> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
> name='mirrormaker2-cluster-configs')], allowAutoTopicCreation=true,
> includeClusterAuthorizedOperations=false,
> includeTopicAuthorizedOperations=false) to node
> nossl-w-kafka-1.nossl-w-kafka-brokers.kafka.svc:9092 (id: 1 rack: null)
> (org.apache.kafka.clients.NetworkClient)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
> 2023-02-23 22:05:54,732 DEBUG [Consumer clientId=consumer-null-5,
> groupId=null] Sending METADATA request with header
> RequestHeader(apiKey=METADATA, apiVersion=11, clientId=consumer-null-5,
> correlationId=17039) and timeout 30000 to node 1:
> MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
> name='mirrormaker2-cluster-offsets'),
> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
> name='mirrormaker2-cluster-status'),
> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='mmtest'),
> MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA,
> name='mirrormaker2-cluster-configs')], allowAutoTopicCreation=true,
> includeClusterAuthorizedOperations=false,
> includeTopicAuthorizedOperations=false)
> (org.apache.kafka.clients.NetworkClient)
> [task-thread-my-source-cluster-west->my-target-cluster-east.MirrorSourceConnector-0]
> ```
>
>
> Intermittently, I'm seeing -
> org.apache.kafka.common.errors.DisconnectException
> when MirrorMaker tries to connect to the broker on the source cluster, even
> though MM2 is installed on the src connection.
>
> Data in the src topic (mmtest on nossl-w, region - us-west1) is not moving
> to target cluster/topic (mmtest on nossl-e, region - us-east1)
>
> Pls note:
> If i logon to the MirrorMaker pod(in cluster - nossl-w), as expected - I'm
> able to access the kafka pods on the nossl-w.
>
> Any ideas on what needs to be done to debug/resolve this issue & enable
> KafkaMirrorMaker2 working ?
>
> tia!
>
>
>
>
>
>
>
>
>
>
>
