Hi As Mark said, scheduler mode works within application ie within a Spark Session and Spark context. This is also clear if you think where you set the configuration - in a Spark Config which used to build a context.
If you are using Yarn as resource manager, however, you can set YARN with fair scheduler. If you do so, both of your applications will get "Fair" treatment from Yarn, ie get resources in round robin manner. If you want your App A to give up resources while using them, you need to set preemption within Yarn and priority of applications so that preemption can kick in. HTH... Best, Ayan On Fri, Jul 21, 2017 at 7:11 AM, Mark Hamstra <m...@clearstorydata.com> wrote: > The fair scheduler doesn't have anything to do with reallocating resource > across Applications. > > https://spark.apache.org/docs/latest/job-scheduling.html# > scheduling-across-applications > https://spark.apache.org/docs/latest/job-scheduling.html# > scheduling-within-an-application > > On Thu, Jul 20, 2017 at 2:02 PM, Gokula Krishnan D <email2...@gmail.com> > wrote: > >> Mark, Thanks for the response. >> >> Let me rephrase my statements. >> >> "I am submitting a Spark application(*Application*#A) with >> scheduler.mode as FAIR and dynamicallocation=true and it got all the >> available executors. >> >> In the meantime, submitting another Spark Application (*Application* >> # B) with the scheduler.mode as FAIR and dynamicallocation=true but it got >> only one executor. " >> >> Thanks & Regards, >> Gokula Krishnan* (Gokul)* >> >> On Thu, Jul 20, 2017 at 4:56 PM, Mark Hamstra <m...@clearstorydata.com> >> wrote: >> >>> First, Executors are not allocated to Jobs, but rather to Applications. >>> If you run multiple Jobs within a single Application, then each of the >>> Tasks associated with Stages of those Jobs has the potential to run on any >>> of the Application's Executors. Second, once a Task starts running on an >>> Executor, it has to complete before another Task can be scheduled using the >>> prior Task's resources -- the fair scheduler is not preemptive of running >>> Tasks. >>> >>> On Thu, Jul 20, 2017 at 1:45 PM, Gokula Krishnan D <email2...@gmail.com> >>> wrote: >>> >>>> Hello All, >>>> >>>> We are having cluster with 50 Executors each with 4 Cores so can avail >>>> max. 200 Executors. >>>> >>>> I am submitting a Spark application(JOB A) with scheduler.mode as FAIR >>>> and dynamicallocation=true and it got all the available executors. >>>> >>>> In the meantime, submitting another Spark Application (JOB B) with the >>>> scheduler.mode as FAIR and dynamicallocation=true but it got only one >>>> executor. >>>> >>>> Normally this situation occurs when any of the JOB runs with the >>>> Scheduler.mode= FIFO. >>>> >>>> 1) Have your ever faced this issue if so how to overcome this?. >>>> >>>> I was in the impression that as soon as I submit the JOB B the Spark >>>> Scheduler should distribute/release few resources from the JOB A and share >>>> it with the JOB A in the Round Robin fashion?. >>>> >>>> Appreciate your response !!!. >>>> >>>> >>>> Thanks & Regards, >>>> Gokula Krishnan* (Gokul)* >>>> >>> >>> >> > -- Best Regards, Ayan Guha
