[
https://issues.apache.org/jira/browse/FLINK-16602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17059924#comment-17059924
]
Canbin Zheng edited comment on FLINK-16602 at 3/19/20, 3:48 AM:
----------------------------------------------------------------
Hi, [~fly_in_gis]. Thanks for the reponse.
> In my original design, the rest service is optional and used for accessing
> the dashboard/rest server outside of the K8s cluster. So only rest port is
> necessary. We always need to expose the rest port on the internal service.
> Because when you have deployer running in the K8s, which will be used to
> start Flink cluster and proxy webui, the internal service will be used to
> forward the webui traffic to jobmanager.
I haven't seen the design for the so-called deployer, do I miss some FLIPs? For
the current design, the internal Service does not need to expose the rest port,
and we could add that port back if we need that, for example, when we introduce
the so-called deployer.
Another concern is that usually, the two Services need different settings, for
example, the external Service should support the users to customize annotations
for LB type, Security Policy, etc. However, the internal one does not need such
kind of features.
Therefore, for the long-term evolution, it would be better to decouple the
internal and external Service.
> We will have the same situation as above when using the deployer. Using the
> service is easier to forward traffic to rest server. Moreover, i think there
> is no harm to leave the internal service there.
So do you mean we forward the TM requests directly to the active JM via IP, but
use the internal Service to help forward rest traffic? Why not always use the
IP of the active JM for internal requests forwarding in the high availability
case?
> Using the headless is good choice to reduce the pressure of kube proxy. If we
> use ipvs, it will not be a urgent problem. Also using headless service will
> cause some dns cache problem when the jobmanager failover. Since hostname do
> not change, the corresponding IP address changes. I am not sure about this,
> but i think there is such risk. So in my original design, i choose ClusterIP
> instead of headless service.
In our production environment, we have met the problem that the whole K8s
cluster hangs since there are too many rules in the iptables, so we propose to
use headless Service if it meets the requirement. So far we do not encounter
the DNS cache problem. For the ipvs setup, indeed it’s not an urgent problem.
was (Author: felixzheng):
Hi, [~fly_in_gis]. Thanks for the reponse.
> In my original design, the rest service is optional and used for accessing
> the dashboard/rest server outside of the K8s cluster. So only rest port is
> necessary. We always need to expose the rest port on the internal service.
> Because when you have deployer running in the K8s, which will be used to
> start Flink cluster and proxy webui, the internal service will be used to
> forward the webui traffic to jobmanager.
I haven't seen the design for the so-called deployer, do I miss some FLIPs? For
the current design, the internal Service does not need to expose the rest port,
and we could add that port back if we need that, for example, when we introduce
the so-called deployer.
Another concern is that usually, the two Services need different settings, for
example, the external Service should support the users to customize annotations
for LB type, Security Policy, etc. However, the internal one does not need such
kind of features.
Therefore, for the long-term evolution, it would be better to decouple the
internal and external Service.
> We will have the same situation as above when using the deployer. Using the
> service is easier to forward traffic to rest server. Moreover, i think there
> is no harm to leave the internal service there.
So do you mean we forward the TM requests directly to the active JM via IP, but
use the internal Service to help forward rest traffic? Does it make sense to
separate these two scenarios? Why not always use the IP of the active JM for
internal requests forwarding in the high availability case?
> Using the headless is good choice to reduce the pressure of kube proxy. If we
> use ipvs, it will not be a urgent problem. Also using headless service will
> cause some dns cache problem when the jobmanager failover. Since hostname do
> not change, the corresponding IP address changes. I am not sure about this,
> but i think there is such risk. So in my original design, i choose ClusterIP
> instead of headless service.
In our production environment, we have met the problem that the whole K8s
cluster hangs since there are too many rules in the iptables, so we propose to
use headless Service if it meets the requirement. So far we do not encounter
the DNS cache problem. For the ipvs setup, indeed it’s not an urgent problem.
> Rework the Service design for Kubernetes deployment
> ---------------------------------------------------
>
> Key: FLINK-16602
> URL: https://issues.apache.org/jira/browse/FLINK-16602
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
> Affects Versions: 1.10.0
> Reporter: Canbin Zheng
> Priority: Major
> Fix For: 1.11.0
>
>
> {color:#0e101a}At the moment we usually create two Services for a Flink
> application, one is the internal Service and the other is the so-called rest
> Service, the previous aims for forwarding request from the TMs to the JM, and
> the rest Service mainly serves as an external service for the Flink
> application. Here is a summary of the issues:{color}
> # {color:#0e101a}The functionality boundary of the two Services is not clear
> enough since the internal Service could also become the rest Service when its
> exposed type is ClusterIP.{color}
> # {color:#0e101a}For the high availability scenario, we create a useless
> internal Service which does not help forward the internal requests since the
> TMs directly communicate with the JM via the IP or hostname of the JM
> Pod.{color}
> # {color:#0e101a}Headless service is enough to help forward the internal
> requests from the TMs to the JM. Service of ClusterIP type would add
> corresponding rules into the iptables, too many rules in the iptables would
> lower the kube-proxy's efficiency in refreshing iptables while notified of
> change events, which could possibly cause severe stability problems in a
> Kubernetes cluster.{color}
>
> {color:#0e101a}Therefore, we propose some improvements to the current
> design:{color}
> # {color:#0e101a}Clarify the functionality boundary for the two Services,
> the internal Service only serves the internal communication from TMs to JM,
> while the rest Service makes the Flink cluster accessible from outside. The
> internal Service only exposes the RPC and BLOB ports while the external one
> exposes the REST port.{color}
> # {color:#0e101a}Do not create the internal Service in the high availability
> case.{color}
> # {color:#0e101a}Use HEADLESS type for the internal Service.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
