Re: Kafka Mirror maker stops replicating

2023-02-23 Thread Arpit Jain
I tried again by increasing the max.poll.interval.ms to a really high number like 12 hrs and since then it has been replicating fine. What I observed was that after every max.poll.interval elapse, it was getting stuck and replication was stopping. Putting a high number like 12 hrs, it worked fine f

MirrorMaker2 on GKE

2023-02-23 Thread karan alang
Hello All, Anyone has installed KafkaMirrorMaker2 on GKE ? Need some help to debug/resolve issues that I've been having. I've 2 clusters on GKE with strimzi kafka installed. cluster1(nossl-e) on use-east1, cluster2(nossl-e) on us-west1. Both the clusters are Auto-pilot cluster (mentioning this,

kafka-producer-perf-test.sh maximum throughput limits for broker on single node

2023-02-23 Thread Tomasz Sąciński
Hello users. I test max throughput to a single kafka broker but results do not match speed limits disk or network. /kafka-producer-perf-test.sh --topic perf-test --num-records 3000 --record-size 100 --throughput -1 --producer-props bootstrap.servers=localhost:9092 3000 records sent, 98.87285

Re: MirrorMaker2 on GKE

2023-02-23 Thread karan alang
Logs with the Disconnect Exception (when i set the loglevel to - INFO) : ``` [SourceTaskOffsetCommitter-1] 2023-02-23 22:54:13,546 INFO [Consumer clientId=consumer-null-4, groupId=null] Error sending fetch request (sessionId=1509420532, epoch=492) to node 2: (org.apache.kafka.clients.FetchSessi

Re: MirrorMaker2 on GKE

2023-02-23 Thread karan alang
I just figured out why there are so many Disconnection exceptions .. the podIP keeps changing in GCP, hence when MirrorMaker tries to access the pod using the older IP, the pod is not found .. hence the DisconnectException However, it is still not clear why the MirrorMaker is not transferring data