zhengcanbin opened a new pull request #11456: [FLINK-16602][k8s] Rework the internal/external Service URL: https://github.com/apache/flink/pull/11456 ## What is the purpose of the change At the moment we usually create two Services for a Flink application, one is the internal Service and the other is the so-called rest Service, the previous aims for forwarding request from the TMs to the JM, and the rest Service mainly serves as an external service for the Flink application. Here is a summary of the issues: 1. The functionality boundary of the two Services is not clear enough since the internal Service could also become the rest Service when its exposed type is ClusterIP. 2. For the high availability scenario, we create a useless internal Service which does not help forward the internal requests since the TMs directly communicate with the JM via the IP or hostname of the JM Pod. 3. Headless service is enough to help forward the internal requests from the TMs to the JM. Service of ClusterIP type would add corresponding rules into the iptables, too many rules in the iptables would lower the kube-proxy's efficiency in refreshing iptables while notified of change events, which could possibly cause severe stability problems in a Kubernetes cluster. Therefore, we propose some improvements to the current design: 1. Clarify the functionality boundary for the two Services, the internal Service only serves the internal communication from TMs to JM, while the rest Service makes the Flink cluster accessible from outside. The internal Service only exposes the RPC and BLOB ports while the external one exposes the REST port. 2. Do not create the internal Service in the high availability case. 3. Use HEADLESS type for the internal Service. ## Verifying this change This change added unit tests and can be verified on a real K8s cluster as follows: 1. Deploy a non-HA session cluster and check there are two dedicated Services created, the internal one has NONE CLUSTER-IP. 2. Deploy an HA session cluster and check only the rest Service is created. 3. Check that the internal Service only exposes the RPC and BLOB port while the external one exposes the REST port. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
