[
https://issues.apache.org/jira/browse/FLINK-15437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17005906#comment-17005906
]
Xintong Song commented on FLINK-15437:
--------------------------------------
I've checked the codes. The problem is that '-D' properties are applied only to
the cluster (JM) configuration but not the to client configuration, while the
client is using the memory size configuration before deploying the cluster.
I'll try to provide a fix soon.
> Start session with property of "-Dtaskmanager.memory.process.size" not work
> ---------------------------------------------------------------------------
>
> Key: FLINK-15437
> URL: https://issues.apache.org/jira/browse/FLINK-15437
> Project: Flink
> Issue Type: Bug
> Components: API / Core
> Affects Versions: 1.10.0
> Reporter: xiaojin.wy
> Priority: Major
> Fix For: 1.10.0
>
>
> *The environment:*
> Yarn session cmd is as below, and the flink-conf.yaml has not the property of
> "taskmanager.memory.process.size":
> export HADOOP_CLASSPATH=`hadoop classpath`;export
> HADOOP_CONF_DIR=/dump/1/jenkins/workspace/Stream-Spark-3.4/env/hadoop_conf_dirs/blinktest2;
> export BLINK_HOME=/dump/1/jenkins/workspace/test/blink_daily;
> $BLINK_HOME/bin/yarn-session.sh -d -qu root.default -nm 'Session Cluster of
> daily_regression_stream_spark_1.10' -jm 1024 -n 20 -s 10
> -Dtaskmanager.memory.process.size=1024m
> *After execute the cmd above, there is a exception like this:*
> 2019-12-30 17:54:57,992 INFO org.apache.hadoop.yarn.client.RMProxy
> - Connecting to ResourceManager at
> z05c07224.sqa.zth.tbsite.net/11.163.188.36:8050
> 2019-12-30 17:54:58,182 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli
> - Error while running the Flink session.
> org.apache.flink.configuration.IllegalConfigurationException: Either Task
> Heap Memory size (taskmanager.memory.task.heap.size) and Managed Memory size
> (taskmanager.memory.managed.size), or Total Flink Memory size
> (taskmanager.memory.flink.size), or Total Process Memory size
> (taskmanager.memory.process.size) need to be configured explicitly.
> at
> org.apache.flink.runtime.clusterframework.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:145)
> at
> org.apache.flink.client.deployment.AbstractClusterClientFactory.getClusterSpecification(AbstractClusterClientFactory.java:44)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:557)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$5(FlinkYarnSessionCli.java:803)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1804)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:803)
> ------------------------------------------------------------
> The program finished with the following exception:
> org.apache.flink.configuration.IllegalConfigurationException: Either Task
> Heap Memory size (taskmanager.memory.task.heap.size) and Managed Memory size
> (taskmanager.memory.managed.size), or Total Flink Memory size
> (taskmanager.memory.flink.size), or Total Process Memory size
> (taskmanager.memory.process.size) need to be configured explicitly.
> at
> org.apache.flink.runtime.clusterframework.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:145)
> at
> org.apache.flink.client.deployment.AbstractClusterClientFactory.getClusterSpecification(AbstractClusterClientFactory.java:44)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:557)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$5(FlinkYarnSessionCli.java:803)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1804)
> at
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:803)
> *The flink-conf.yaml is :*
> jobmanager.rpc.address: localhost
> jobmanager.rpc.port: 6123
> jobmanager.heap.size: 1024m
> taskmanager.memory.total-process.size: 1024m
> taskmanager.numberOfTaskSlots: 1
> parallelism.default: 1
> jobmanager.execution.failover-strategy: region
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
