If you run that job then the driver will ALWAYS run in the machine from where you are issuing the spark submit command (E.g. some edge node with the clients installed). No matter where the resource manager is running.
If you change yarn-client for yarn-cluster then your driver will start somewhere else in the cluster as will the workers and the spark submit command will return before the program finishes On Tue, 7 Jun 2016, 14:53 Jacek Laskowski, <ja...@japila.pl> wrote: > Hi, > > --master yarn-client is deprecated and you should use --master yarn > --deploy-mode client instead. There are two deploy-modes: client > (default) and cluster. See > http://spark.apache.org/docs/latest/cluster-overview.html. > > Pozdrawiam, > Jacek Laskowski > ---- > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Tue, Jun 7, 2016 at 2:50 PM, Mich Talebzadeh > <mich.talebza...@gmail.com> wrote: > > ok thanks > > > > so I start SparkSubmit or similar Spark app on the Yarn resource manager > > node. > > > > What you are stating is that Yan may decide to start the driver program > in > > another node as opposed to the resource manager node > > > > ${SPARK_HOME}/bin/spark-submit \ > > --driver-memory=4G \ > > --num-executors=5 \ > > --executor-memory=4G \ > > --master yarn-client \ > > --executor-cores=4 \ > > > > Due to lack of resources in the resource manager node? What is the > > likelihood of that. The resource manager node is the defector master > node in > > all probability much more powerful than other nodes. Also the node that > > running resource manager is also running one of the node manager as > well. So > > in theory may be in practice may not? > > > > HTH > > > > Dr Mich Talebzadeh > > > > > > > > LinkedIn > > > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > > > > > > > http://talebzadehmich.wordpress.com > > > > > > > > > > On 7 June 2016 at 13:20, Sebastian Piu <sebastian....@gmail.com> wrote: > >> > >> What you are explaining is right for yarn-client mode, but the question > is > >> about yarn-cluster in which case the spark driver is also submitted and > run > >> in one of the node managers > >> > >> > >> On Tue, 7 Jun 2016, 13:45 Mich Talebzadeh, <mich.talebza...@gmail.com> > >> wrote: > >>> > >>> can you elaborate on the above statement please. > >>> > >>> When you start yarn you start the resource manager daemon only on the > >>> resource manager node > >>> > >>> yarn-daemon.sh start resourcemanager > >>> > >>> Then you start nodemanager deamons on all nodes > >>> > >>> yarn-daemon.sh start nodemanager > >>> > >>> A spark app has to start somewhere. That is SparkSubmit. and that is > >>> deterministic. I start SparkSubmit that talks to Yarn Resource Manager > that > >>> initialises and registers an Application master. The crucial point is > Yarn > >>> Resource manager which is basically a resource scheduler. It optimizes > for > >>> cluster resource utilization to keep all resources in use all the time. > >>> However, resource manager itself is on the resource manager node. > >>> > >>> Now I always start my Spark app on the same node as the resource > manager > >>> node and let Yarn take care of the rest. > >>> > >>> Thanks > >>> > >>> Dr Mich Talebzadeh > >>> > >>> > >>> > >>> LinkedIn > >>> > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > >>> > >>> > >>> > >>> http://talebzadehmich.wordpress.com > >>> > >>> > >>> > >>> > >>> On 7 June 2016 at 12:17, Jacek Laskowski <ja...@japila.pl> wrote: > >>>> > >>>> Hi, > >>>> > >>>> It's not possible. YARN uses CPU and memory for resource constraints > and > >>>> places AM on any node available. Same about executors (unless data > locality > >>>> constraints the placement). > >>>> > >>>> Jacek > >>>> > >>>> On 6 Jun 2016 1:54 a.m., "Saiph Kappa" <saiph.ka...@gmail.com> wrote: > >>>>> > >>>>> Hi, > >>>>> > >>>>> In yarn-cluster mode, is there any way to specify on which node I > want > >>>>> the driver to run? > >>>>> > >>>>> Thanks. > >>> > >>> > > >
