Hi Averell, According to the AWS documentation [1], the master node only runs the YARN ResourceManager and the HDFS NameNode. Containers can only by launched on nodes that are running the YARN NodeManager [2]. Therefore, if you want TMs or JMs to be launched on your EMR master node, you have to start the NodeManager process there but I do not know how well this is supported by AWS EMR.
You can choose a smaller server for the master node but keep in mind that it is running the HDFS NameNode as well. The hardware requirements will therefore partially depend on the HDFS workload. Best, Gary [1] https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-instances.html [2] https://hadoop.apache.org/docs/r2.8.0/hadoop-yarn/hadoop-yarn-site/NodeManager.html On Mon, Sep 17, 2018 at 5:22 AM, Averell <lvhu...@gmail.com> wrote: > Hello everyone, > > I'm trying to run Flink on AWS EMR following the guides from Flink doc > <https://ci.apache.org/projects/flink/flink-docs- > master/ops/deployment/yarn_setup.html#run-a-single-flink- > job-on-hadoop-yarn> > and from AWS > <https://docs.aws.amazon.com/emr/latest/ReleaseGuide/flink-configure.html> > > , and it looks like the EMR master is never used, neither for JM nor TM. > "bin/yarn-session.sh -q" only shows the core nodes. We are only running > Flink on that EMR, so it is wasting of resources. > > So, is there any way to use the master node for the job, at least for the > JM > only? > > If that is not possible, should I have different hardware configurations > between the master node and core nodes (smaller server for the master)? > > Thanks and best regards, > Averell > > > > > -- > Sent from: http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/ >
