Hi Paul, I don't think you need 10 GB of heap space for the JobManager. Usually 1 GB are sufficient. Since you have 3 nodes, I would start Flink with 3 task managers. I think you can also launch such a cluster: ./flink-0.9.0/bin/yarn-session.sh -n 3 -jm 1024 -tm 13000
Regarding the memory you are able to use in the end: Initially, you request 10240MB. >From that, we use add a 25% safety margin to avoid that YARN is going to kill the JVM. 10240*0.75 = 7680 MB. So Flink's TaskManager will see 7680 MB when starting up. Flink's Memory manager is only using 70% of the available heap space for managed memory: 7680*0.7 = 5376 MB. The safety margin for YARN is very conservative. As Till already said, you can set a different value for the "yarn.heap-cutoff-ratio" (try 0.15) and see if your job still runs. On Tue, Jul 14, 2015 at 10:47 AM, Pa Rö <paul.roewer1...@googlemail.com> wrote: > hello community, > > i want run my flink app on a cluster (cloudera 5.4.4) with 3 nodes (one pc > has i7 8core with 16GB RAM). now i want submit my spark job on yarn (20GB > RAM). > > my script to deploy the flink cluster on yarn: > > export HADOOP_CONF_DIR=/etc/hadoop/conf/ > ./flink-0.9.0/bin/yarn-session.sh -n 1 -jm 10240 -tm 10240 > > my script to submit the job is to time the following: > > ./flink-0.9.0/bin/flink run /home/marcel/Desktop/ma-flink.jar > > in the flink dashbord shown are only 5GB memory used for computed my job? > > maybe my configuration is not the optimal?? > > best regards, > paul >
