Re: Kubernetes JobManager and TaskManager minimum/maximum resources

te minimum >> amount of CPU and RAM that is required to run the JobManager and >> TaskManager processes? >> >> Some of the example deployment yaml examples have CPU set at 1 full vCPU >> and memory at 2GB (2048 MB). If you factor in JobManager HA, and 1 or more >> TaskM

Re: Kubernetes JobManager and TaskManager minimum/maximum resources

? :) Thanks Gyula On Fri, 20 Jan 2023 at 05:32, Lee Parayno wrote: > For application mode FlinkDeployments (maybe even session mode) in > Kubernetes from the Flink Kubernetes Operator what is the absolute minimum > amount of CPU and RAM that is required to run the JobManager and >

Kubernetes JobManager and TaskManager minimum/maximum resources

For application mode FlinkDeployments (maybe even session mode) in Kubernetes from the Flink Kubernetes Operator what is the absolute minimum amount of CPU and RAM that is required to run the JobManager and TaskManager processes? Some of the example deployment yaml examples have CPU set at 1

Re: How JobManager and TaskManager find each other?

Hi KristoffSC, Regarding your questions inline. > 1. task deployment descriptor The `TaskDeploymentDescriptor` is used by JobMaster to submit a task to TaskManager. Since the JobMaster knows all the operator and its location, it will put the upstream operator location in the `TaskDeploymentDescri

Re: How JobManager and TaskManager find each other?

Thanks about clarification for NAT, Moving NAT issue aside for a moment", Is the process of sending "task deployment descriptor" that you mentioned in "Feb 26, 2020; 4:18pm" a specially the process of notifying TaskManager about IP of participating TaskManagers in job described somewhere? I'm fam

Re: How JobManager and TaskManager find each other?

Well i think the biggest challenge when running Flink on PaaS platforms is about the NAT network. Since the jobmanager/taskmanagers are running in a NAT network. So they could not talk directly. What we need to do is make the JM/TM to bind the local machine hostname and use a bind-port for the comm

Re: How JobManager and TaskManager find each other?

/So do you mean the ip address changes during running or the taskmanager failed and relaunched with a same hostname, but the ip address is different?/ Well that also but actually I was thinking about running FLink on PaaS platforms where process can be re-spawned during runtime on different machin

Re: How JobManager and TaskManager find each other?

So do you mean the ip address changes during running or the taskmanager failed and relaunched with a same hostname, but the ip address is different? I am not sure the TM registers to JM with hostname or ip address. We could confirm that from @Zhijiang. For the NAT, currently Flink could not work

Re: How JobManager and TaskManager find each other?

Thank you very much, what about if node Ip changes? Does it also supports dns or "raw" IP addresses only. I'm thinking about cloud deployments where actual service/process can be rescheduled to a different box but there is name resolving mechanism. Also what if there is NAT between Task Manager a

Re: How JobManager and TaskManager find each other?

ld be sent to the required peers during task schedule and deployment. Best, Zhijiang -- From:KristoffSC Send Time:2020 Feb. 26 (Wed.) 19:39 To:user Subject:Re: How JobManager and TaskManager find each other? Thanks all for the an

Re: How JobManager and TaskManager find each other?

Thanks all for the answers, One more question though. In [1] we can see that task managers are talking with each other - sending data streams. How each task manager knows the address of other task managers? [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/concepts/runtime.html#jo

Re: How JobManager and TaskManager find each other?

Hi Krzysztof, Xintong has share some information about the configuration. I just want to provide some additional information. >> How JobManager and TaskManager find each other? For non-HA Flink cluster, the taskmanager should have the pre-configured jobmanager address. For HA Flink cluste

Re: How JobManager and TaskManager find each other?

different JM address/ports in your TM configurations, so the TM knows which JM to connect to. Thank you~ Xintong Song On Fri, Feb 21, 2020 at 6:38 AM KristoffSC wrote: > Hi all, > I was wondering how JobManager and TaskManager find each other? > Do they use multicast for this? >

How JobManager and TaskManager find each other?

Hi all, I was wondering how JobManager and TaskManager find each other? Do they use multicast for this? Can it be configure to use domain names instead IP's? What I have to do to have two Flink Clusters in same IP network? How I should start task manager in order to tell him, to conne

Re: How to access JobManager and TaskManager

Hi Tim, "job_env" is a variable I passed to launch YARN application. I just want to access it in my flink application main method. There is is no documentation on how to access customized job environment variables or settings. Thanks, Tao -- Sent from: http://apache-flink-user-mailing-list-arc

Re: How to access JobManager and TaskManager

I don't have this property in my local running Flink cluster. Which Flink version and deployment are you using? Are you sure this property is not set in your flink-conf.yaml? Regards, Timo Am 1/31/18 um 7:51 AM schrieb xiatao123: In the web UI, I can see these information under JobManager.

How to access JobManager and TaskManager

In the web UI, I can see these information under JobManager. How can I access variables job_env in main method? Job Manager Configuration env.hadoop.conf.dir /etc/hadoop/conf env.yarn.conf.dir /etc/hadoop/conf high-availability.cluster-idapplication_1517362137681_0001 job_env stage j

Re: Jobmanager and Taskmanager

Yes, a given host may be used as both a job manager and as a task manager. On Mon, Sep 11, 2017 at 8:27 AM, AndreaKinn wrote: > Hi, > I'm configuring a cluster composed by just three nodes. > Looking at cluster setup guide > release-0.8/cluste

Jobmanager and Taskmanager

Hi, I'm configuring a cluster composed by just three nodes. Looking at cluster setup guide I'm setting in the jobmanager the addresses of the workers. A jobmanager can be used also as an additional taskmanager? i.

Re: Running the JobManager and TaskManager on the same node in a cluster

Hi Dominik, Your observation is right, running the JobManager and TaskManager on the same node is no problem. If that machine fails, both services will be affected, but as long as you have infrastructure in place (YARN for example) to start them somewhere else, nothing bad will happen. Regarding

Running the JobManager and TaskManager on the same node in a cluster

Hi, It is generally recommended for streaming engines, also including Flink to run a separate master node - in the case of Flink, the JobManager. However, why should one in Flink run the JobManager on a separate node? Performance wise, the JobManager isn’t intense unlike of course TaskManager