Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

2021-02-03 Thread sidhant gupta
Thanks Yang for your help. On Thu, Feb 4, 2021, 8:28 AM Yang Wang wrote: > Yes, if you are using the CLI(e.g. flink run/list/cancel -t yarn-session > ...) for the job management, > it will eventually call the RestClusterClient, which could retrieve the > leader JobManager address from ZK. > > Pl

Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

2021-02-03 Thread Yang Wang
Yes, if you are using the CLI(e.g. flink run/list/cancel -t yarn-session ...) for the job management, it will eventually call the RestClusterClient, which could retrieve the leader JobManager address from ZK. Please ensure that you have specified the HA related config options in CLI via -D or set

Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

2021-02-03 Thread sidhant gupta
Is it possible to use flink CLI instead of flink client for connecting zookeeper using network load balancer to retrieve the leader Jobmanager address? On Wed, Feb 3, 2021, 12:42 PM Yang Wang wrote: > I think the Flink client could make a connection with ZooKeeper via the > network load balancer

Re: Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

2021-02-02 Thread Yang Wang
I think the Flink client could make a connection with ZooKeeper via the network load balancer. Flink client is not aware of whether it is a network balancer or multiple ZooKeeper server address. After then Flink client will retrieve the active leader JobManager address via ZooKeeperHAService and su

Flink cli to upload flink jar and run flink jobs in the Jenkins pipeline

2021-02-02 Thread sidhant gupta
Hi I have a flink ECS cluster setup with HA mode using zookeeper where I have 2 jobmanagers out of which one of will be elected as leader using zookeeper leader election. I have one application load balancer in front of the jobmanagers and one network load balancer in front of zookeeper. As per [