Yes. I agree with you that it is valuable to document how to start multiple
JobManagers in HA mode for K8s deployment.
I have a ticket to track this documentation improvement[1].

[1]. https://issues.apache.org/jira/browse/FLINK-21082

Best,
Yang

Ufuk Celebi <u...@apache.org> 于2021年1月21日周四 下午9:08写道:

> @Yang: I think this would be valuable to document. I think it's a natural
> question to ask whether you can have standby JMs with Kubernetes. What do
> you think? If you agree, we could create a JIRA ticket and work on the
> "official" docs for this.
>
> On Thu, Jan 21, 2021, at 5:05 AM, Yang Wang wrote:
>
> Hi Amit Bhatia
>
> > What is the correct way to start three jobmanager replicas with zk? Is
> there any link which explains this deployment scenario and configuration?
> Please find more information in the last mail. Unfortunately, we do not
> have some documentation to guide the users how to achieve that.
>
> > How we'll identify that out of three replicas, which Job Manager replica
> is the leader?
> Just like what I have said, using a K8s service for the jobmanager rpc
> address is not a good practice.
> TaskManager/Client could not know which replica is the leader. Instead, we
> should bind the rpc address
> to pod ip. After then, TaskManager/Client could find the leader
> address(pod ip) via ZooKeeper.
>
> Could you please update your yaml files and deploy again? I think you will
> have different results then.
>
> Best,
> Yang
>
> Yang Wang <danrtsey...@gmail.com> 于2021年1月21日周四 上午11:59写道:
>
> Hi Chirag Dewan,
>
> Yes, we could have multiple replicas with ZK HA in K8 as well. Multiple
> JobManagers will contend for
> a leader and then write its rpc address to the ZooKeeper nodes. You could
> find more information how the
> HA service works here[1]. It is about the KubernetesHAService, but the
> ZooKeeperHAService has the same
> mechanism.
>
> In such a case, I strongly suggest not using the service as the JobManager
> rpc address. Otherwise, we
> will have the issue you have mentioned. There are 3 replicas behind the
> same service endpoint and only
> one of them is the leader. TaskManager/Client do not know how to contact
> the leader.
>
> Instead, I suggest not creating the internal service and bind the pod ip
> to the JobManager rpc address.
> After then, TaskManager/Client will retrieve the leader address(pod ip +
> port) and contact via such an address.
>
> Please find more information and the example here[1].
>
> [1].
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-144%3A+Native+Kubernetes+HA+for+Flink
> [2].
> https://issues.apache.org/jira/browse/FLINK-20982?focusedCommentId=17265715&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17265715
>
> Best,
> Yang
>
>
> Amit Bhatia <bhatia.amit1...@gmail.com> 于2021年1月20日周三 下午12:27写道:
>
>   Hi Yang,
>
> I tried the deployment of flink with three replicas of Jobmanger to test a
> faster job recovery scenario.  Below is my deployment :
>
>  $ kubectl get po -namit | grep zk
> eric-data-coordinator-zk-0                                        1/1
> Running            0          6d21h
> eric-data-coordinator-zk-1                                        1/1
> Running            0          6d21h
> eric-data-coordinator-zk-2                                        1/1
> Running            0          6d21h
> flink-jobmanager-ha-zk-1-5d58dc469-8bjpb                          1/1
> Running            0          19h
> flink-jobmanager-ha-zk-1-5d58dc469-klg5p                          1/1
> Running            0          19h
> flink-jobmanager-ha-zk-1-5d58dc469-kvwzk                          1/1
> Running            0          19h
>
>
>  $ kubectl get svc -namit | grep zk
> flink-jobmanager-ha-rest-zk1                NodePort       10.100.118.186
>   <none>        8081:32115/TCP                                 21h
> flink-jobmanager-ha-zk1                     ClusterIP      10.111.135.174
>   <none>        6123/TCP,6124/TCP,8081/TCP                     21h
> eric-data-coordinator-zk                    ClusterIP      10.105.139.167
>   <none>        2181/TCP,8080/TCP,21007/TCP                    7d20h
> eric-data-coordinator-zk-ensemble-service   ClusterIP      None
>   <none>        2888/TCP,3888/TCP                              7d20h
>
> Flink Configmap:
> ====================
> apiVersion: v1
> kind: ConfigMap
> metadata:
>   name: flink-config-ha-zk-1
>   namespace: amit
>   labels:
>     app: flink
> data:
>   flink-conf.yaml: |+
>     jobmanager.rpc.address: flink-jobmanager-ha-zk1
>     taskmanager.numberOfTaskSlots: 2
>     blob.server.port: 6124
>     jobmanager.rpc.port: 6123
>     taskmanager.rpc.port: 6122
>     queryable-state.proxy.ports: 6125
>     jobmanager.memory.process.size: 1600m
>     taskmanager.memory.process.size: 1728m
>     parallelism.default: 2
>     # High Availability parameters
>     high-availability: zookeeper
>     high-availability.cluster-id: /haclusterzk1
>     high-availability.storageDir: file:///opt/flink/recovery/
>     high-availability.zookeeper.path.root: /flinkhazk
>     high-availability.zookeeper.quorum: eric-data-coordinator-zk:2181
>     high-availability.jobmanager.port: 6123
> ===============================================================
>
> Out of the three replicas of Job manager pods in one of the pod i am
> getting this error:
>
> 2021-01-19 08:18:33,982 INFO
>  org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService []
> - Starting ZooKeeperLeaderElectionService
> ZooKeeperLeaderElectionService{leaderPath='/leader/resource_manager_lock'}.
> 2021-01-19 08:21:39,381 WARN
>  org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever []
> - Error while retrieving the leader gateway. Retrying to connect to
> akka.tcp://flink@flink-jobmanager-ha-zk1:6123/user/rpc/dispatcher_1.
> 2021-01-19 08:21:42,521 WARN
>  org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever []
> - Error while retrieving the leader gateway. Retrying to connect to
> akka.tcp://flink@flink-jobmanager-ha-zk1:6123/user/rpc/dispatcher_1.
> 2021-01-19 08:21:45,508 WARN
>  org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever []
> - Error while retrieving the leader gateway. Retrying to connect to
> akka.tcp://flink@flink-jobmanager-ha-zk1:6123/user/rpc/dispatcher_1.
> 2021-01-19 08:21:46,369 WARN
>  org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever []
> - Error while retrieving the leader gateway. Retrying to connect to
> akka.tcp://flink@flink-jobmanager-ha-zk1:6123/user/rpc/dispatcher_1.
> 2021-01-19 08:22:13,658 WARN
>  org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever []
> - Error while retrieving the leader gateway. Retrying to connect to
> akka.tcp://flink@flink-jobmanager-ha-zk1:6123/user/rpc/dispatcher_1.
> 2021-01-20 04:10:39,836 WARN
>  org.apache.flink.runtime.webmonitor.retriever.impl.RpcGatewayRetriever []
> - Error while retrieving the leader gateway. Retrying to connect to
> akka.tcp://flink@flink-jobmanager-ha-zk1:6123/user/rpc/dispatcher_1.
>
>
> And when trying to access the GUI getting below error:
>
> [image: image.png]
>
> In zookeeper i could see all the three id's are there
>
> [zk: localhost:2181(CONNECTED) 5] ls
> /flinkhazk/haclusterzk1/leaderlatch/dispatcher_lock
> [_c_1d5fc8b1-063f-4a1c-ad0f-ec46b6f10f36-latch-0000000020,
> _c_229d0739-8854-4a5a-ace7-377d9edc575f-latch-0000000018,
> _c_4eac3aaf-3f0f-4297-ac7f-086821548697-latch-0000000019]
> [zk: localhost:2181(CONNECTED) 6]
>
> So i have below queries on this:
>
> 1) what is the correct way to start three jobmanager replicas with zk ? Is
> there any link which explains this deployment scenario and configuration ?
>
> 2) How we'll identify that out of three replicas, which Job Manager
> replica is the leader ?
>
> Regards,
> Amit Bhatia
>
>
>
> On Wed, Jan 20, 2021 at 9:44 AM Chirag Dewan <chirag.dewa...@yahoo.in>
> wrote:
>
>
> Hi,
>
> Can we have multiple replicas with ZK HA in K8 as well?
> In this case, how does Task Managers and clients recover the Job Manager
> RPC address? Are they updated in ZK?
> Also, since there are 3 replicas behind the same service endpoint and only
> one of them is the leader, how should clients reach the leader Job Manager?
>
> On Wednesday, 20 January, 2021, 07:41:20 am IST, Yang Wang <
> danrtsey...@gmail.com> wrote:
>
>
> If you do not want to run multiple JobManagers simultaneously, then I
> think the "Job" for application cluster
> with HA enable is enough.
> K8s will also launch a new pod/container when the old one terminated
> exceptionally.
>
> Best,
> Yang
>
> Yang Wang <danrtsey...@gmail.com> 于2021年1月20日周三 上午10:08写道:
>
> Yes. Using a "Deployment" instead of "Job" for the application cluster
> also makes sense.
> Actually, in the native K8s integration, we always use the deployment for
> JobManager.
>
> But please note that the deployment may relaunch the JobManager pod even
> though you cancel
> the Flink job.
>
> Best,
> Yang
>
> Ashish Nigam <ashnigamt...@gmail.com> 于2021年1月20日周三 上午5:29写道:
>
> Yang,
> For Application clusters, does it make sense to deploy JobManager as
> "Deployment" rather than as a "Job", as suggested in docs?
> I am asking this because I am thinking of deploying a job manager in HA
> mode even for application clusters.
>
> Thanks
> Ashish
>
>
> On Tue, Jan 19, 2021 at 6:16 AM Yang Wang <danrtsey...@gmail.com> wrote:
>
> Usually, you do not need to start multiple JobManager simultaneously. The
> JobManager is a deployment.
> A new one pod/container will be launched once it terminated exceptionally.
>
> If you still want to start multiple JobManagers to get a faster recovery,
> you could set the replica greater than 1
> for standalone cluster on K8s[1]. For native integration[2], we still have
> not supported such configuration[2].
>
> Please note that the key point to enable HA is not start multiple
> JobManagers simultaneously or sequently.
> You need to set the ZooKeeperHAService[4] or KubernetesHAService[5] to
> ensure the Flink job could recover
> from latest successful checkpoint.
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/standalone/kubernetes.html#session-cluster-resource-definitions
> [2].
> https://ci.apache.org/projects/flink/flink-docs-master/deployment/resource-providers/native_kubernetes.html
> [3]. https://issues.apache.org/jira/browse/FLINK-17707
> [4].
> https://ci.apache.org/projects/flink/flink-docs-master/deployment/ha/zookeeper_ha.html
> [5].
> https://ci.apache.org/projects/flink/flink-docs-master/deployment/ha/kubernetes_ha.html
>
> Best,
> Yang
>
> Amit Bhatia <bhatia.amit1...@gmail.com> 于2021年1月19日周二 下午8:45写道:
>
> Hi,
>
> I am deploying Flink 1.12 on K8s. Can anyone confirm if we can deploy
> multiple job manager pods in K8s for HA or it should always be only a
> single job manager pod ?
>
> Regards,
> Amit Bhatia
>
>
>

Reply via email to