Hmm, so when I submit an application with `spark-submit`, I need to guarantee it resources using YARN queues and not Spark's scheduler pools. Is that correct?
When are Spark's scheduler pools relevant/useful in this context? On Wed, Apr 5, 2017 at 3:54 PM Mark Hamstra <m...@clearstorydata.com> wrote: > grrr... s/your/you're/ > > On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra <m...@clearstorydata.com> > wrote: > > Your mixing up different levels of scheduling. Spark's fair scheduler > pools are about scheduling Jobs, not Applications; whereas YARN queues with > Spark are about scheduling Applications, not Jobs. > > On Wed, Apr 5, 2017 at 12:27 PM, Nick Chammas <nicholas.cham...@gmail.com> > wrote: > > I'm having trouble understanding the difference between Spark fair > scheduler pools > <https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools> > and YARN queues > <https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>. > Do they conflict? Does one override the other? > > I posted a more detailed question about an issue I'm having with this on > Stack Overflow: http://stackoverflow.com/q/43239921/877069 > > Nick > > > ------------------------------ > View this message in context: Spark fair scheduler pools vs. YARN queues > <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-fair-scheduler-pools-vs-YARN-queues-tp28572.html> > Sent from the Apache Spark User List mailing list archive > <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. > > > >
