Hi Gary,
Thank you so much for the detailed explanation and links. Extremely
helpful. For all others interested, this is also available through the YARN
CLI command `yarn application -status {appId}`.
Once again, thanks for your help!
Austin
On Fri, Sep 7, 2018, 2:24 AM Gary Yao <g...@data-artisans.com> wrote:
> Hi Austin,
>
> The config options rest.port, jobmanager.web.port, etc. are intentionally
> ignored on YARN. The port should be chosen randomly to avoid conflicts with
> other containers [1]. I do not see a way how you can set a fixed port at
> the
> moment but there is a related ticket for that [2]. The Flink CLI determines
> the hostname and port from the YARN ApplicationReport [3][4] – you can do
> the
> same.
>
> Best,
> Gary
>
> [1]
> https://github.com/apache/flink/blob/d036417985d3e2b1ca63909007db9710e842abf4/flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnEntrypointUtils.java#L103
>
> [2] https://issues.apache.org/jira/browse/FLINK-5758
>
> [3]
> https://github.com/apache/flink/blob/d036417985d3e2b1ca63909007db9710e842abf4/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L387
>
> [4]
> https://hadoop.apache.org/docs/r2.8.3/api/org/apache/hadoop/yarn/api/records/ApplicationReport.html#getRpcPort()
>
> On Fri, Sep 7, 2018 at 12:33 AM, Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I'm running a YARN session on a cluster with one master and one core and
>> would like to use the Monitoring API programmatically to submit jobs. I
>> have found that the configuration variables are read but ignored when
>> starting the session - it seems to choose a random port each run.
>>
>> Here's a snippet from the startup logs:
>>
>> 2018-09-06 21:44:38,763 INFO
>> org.apache.flink.configuration.GlobalConfiguration - Loading
>> configuration property: env.yarn.conf.dir, /etc/hadoop/conf
>> 2018-09-06 21:44:38,764 INFO
>> org.apache.flink.configuration.GlobalConfiguration - Loading
>> configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
>> 2018-09-06 21:44:38,765 INFO
>> org.apache.flink.configuration.GlobalConfiguration - Loading
>> configuration property: rest.port, 44477
>> 2018-09-06 21:44:38,765 INFO
>> org.apache.flink.configuration.GlobalConfiguration - Loading
>> configuration property: jobmanager.web.port, 44477
>> 2018-09-06 21:44:38,765 INFO
>> org.apache.flink.configuration.GlobalConfiguration - Loading
>> configuration property: high-availability.jobmanager.port, 44477
>> 2018-09-06 21:44:38,775 INFO
>> org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn
>> properties file under /tmp/.yarn-properties-hadoop.
>> 2018-09-06 21:44:39,615 WARN org.apache.hadoop.util.NativeCodeLoader
>> - Unable to load native-hadoop library for your
>> platform... using builtin-java classes where applicable
>> 2018-09-06 21:44:39,799 INFO
>> org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user
>> set to hadoop (auth:SIMPLE)
>> 2018-09-06 21:44:40,045 INFO org.apache.hadoop.yarn.client.RMProxy
>> - Connecting to ResourceManager at
>> ip-10-2-3-71.ec2.internal/10.2.3.71:8032
>> 2018-09-06 21:44:40,312 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster
>> specification: ClusterSpecification{masterMemoryMB=1024,
>> taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=1}
>> 2018-09-06 21:44:43,564 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting
>> application master application_1536250520330_0007
>> 2018-09-06 21:44:43,802 INFO
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
>> application application_1536250520330_0007
>> 2018-09-06 21:44:43,802 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for
>> the cluster to be allocated
>> 2018-09-06 21:44:43,804 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying
>> cluster, current state ACCEPTED
>> 2018-09-06 21:44:48,326 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN
>> application has been deployed successfully.
>> 2018-09-06 21:44:48,326 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink
>> YARN client has been started in detached mode. In order to stop Flink on
>> YARN, use the following command or a YARN web interface to stop it:
>> yarn application -kill application_1536250520330_0007
>> Please also note that the temporary files of the YARN session in the home
>> directory will not be removed.
>> 2018-09-06 21:44:48,821 INFO org.apache.flink.runtime.rest.RestClient
>> - Rest client endpoint started.
>> Flink JobManager is now running on ip-10-2-3-25.ec2.internal:38683 with
>> leader id 00000000-0000-0000-0000-000000000000.
>> JobManager Web Interface: http://ip-10-2-3-25.ec2.internal:38683
>>
>>
>> I'm setting both the rest.port and jobmanager.web.port, but both are
>> ignored. Has anyone seen this before?
>>
>> Thanks!
>>
>
>
