Thanks Mark, custom configuration file would be better for me. Changing code will make it affect all the applications, this is too risky for me.
On Wed, Jan 6, 2016 at 10:50 AM, Mark Hamstra <m...@clearstorydata.com> wrote: > The other way to do it is to build a custom version of Spark where you > have changed the value of DEFAULT_SCHEDULING_MODE -- and if you were > paying close attention, I accidentally let it slip that that is what I've > done. I previously wrote "schedulingMode = DEFAULT_SCHEDULING_MODE -- > i.e. SchedulingMode.FAIR", but that should actually be SchedulingMode.FIFO > if you haven't changed the code: > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L65 > > On Tue, Jan 5, 2016 at 5:29 PM, Jeff Zhang <zjf...@gmail.com> wrote: > >> Right, I can override the root pool in configuration file, Thanks Mark. >> >> On Wed, Jan 6, 2016 at 8:45 AM, Mark Hamstra <m...@clearstorydata.com> >> wrote: >> >>> Just configure <pool name="default"> with >>> <schedulingMode>FAIR</schedulingMode> in fairscheduler.xml (or >>> in spark.scheduler.allocation.file if you have over-riden the default name >>> for the config file.) `buildDefaultPool()` will only build the pool named >>> "default" with the default properties (such as schedulingMode = >>> DEFAULT_SCHEDULING_MODE -- i.e. SchedulingMode.FAIR) if that pool name is >>> not already built ( >>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala#L90 >>> ). >>> >>> >>> On Tue, Jan 5, 2016 at 4:15 PM, Jeff Zhang <zjf...@gmail.com> wrote: >>> >>>> Sorry, I don't make it clearly. What I want is the default pool is fair >>>> scheduling. But seems if I want to use fair scheduling now, I have to set >>>> spark.scheduler.pool explicitly. >>>> >>>> On Wed, Jan 6, 2016 at 2:03 AM, Mark Hamstra <m...@clearstorydata.com> >>>> wrote: >>>> >>>>> I don't understand. If you're using fair scheduling and don't set a >>>>> pool, the default pool will be used. >>>>> >>>>> On Tue, Jan 5, 2016 at 1:57 AM, Jeff Zhang <zjf...@gmail.com> wrote: >>>>> >>>>>> >>>>>> It seems currently spark.scheduler.pool must be set as >>>>>> localProperties (associate with thread). Any reason why >>>>>> spark.scheduler.pool can not be used globally. My scenario is that I >>>>>> want >>>>>> my thriftserver started with fair scheduler as the default pool without >>>>>> using set command to set the pool. Is there anyway to do that ? Or do I >>>>>> miss anything here ? >>>>>> >>>>>> -- >>>>>> Best Regards >>>>>> >>>>>> Jeff Zhang >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards >>>> >>>> Jeff Zhang >>>> >>> >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > -- Best Regards Jeff Zhang
