I have the following setup: My Kafka cluster and applications are currently on-premise. Kafka runs on a VM, while my applications are deployed in an on-premise Kubernetes cluster. I am in the process of migrating my applications to GKE, but the Kafka cluster will remain on-premise for now.
When I migrate my applications to GKE and run them there, I experience a significant slowdown when building the GlobalKTable. On-premise, it takes about 6 minutes to build the PVC, but in GKE, it takes 1 hour and 10 minutes. As mentioned, the data is always fetched from the same on-premise Kafka cluster. This slowdown occurs only during the initial build of the GlobalKTable. Other applications that use state stores are also slower (about 3x slower), but this can be explained by the increased network latency between GKE and the on-prem Kafka cluster. However, this does not cause significant issues. Only the GlobalKTable initialization is severely impacted. I have already tried adjusting all possible consumer parameters, but none of the changes made any difference. Has anyone experienced a similar issue or has an idea what could be causing this?
