If I understand what you're trying to do correctly, I think you really just
want one pool, but you want to change the mode *within* the pool to be FAIR
as well
https://spark.apache.org/docs/latest/job-scheduling.html#configuring-pool-properties
you'd still need to change the conf file to set up t
No, these pools are not created per job but per parfor worker and
thus, used to execute many jobs. For all scripts with a single
top-level parfor this is equivalent to static initialization. However,
yes we create these pools dynamically on demand to avoid unnecessary
initialization and handle scen
well, the point was "in a programmatic way without the need for
additional configuration files which is a hassle for a library" -
anyway, I appreciate your comments.
Regards,
Matthias
On Sat, Apr 7, 2018 at 3:43 PM, Mark Hamstra wrote:
>> Providing a way to set the mode of the default scheduler
>
> Providing a way to set the mode of the default scheduler would be awesome.
That's trivial: Just use the pool configuration XML file and define a pool
named "default" with the characteristics that you want (including
schedulingMode FAIR).
You only get the default construction of the pool name
Sorry, but I'm still not understanding this use case. Are you somehow
creating additional scheduling pools dynamically as Jobs execute? If so,
that is a very unusual thing to do. Scheduling pools are intended to be
statically configured -- initialized, living and dying with the
Application.
On Sat
Thanks for the clarification Imran - that helped. I was mistakenly
assuming that these pools are removed via weak references, as the
ContextCleaner does for RDDs, broadcasts, and accumulators, etc. For
the time being, we'll just work around it, but I'll file a
nice-to-have improvement JIRA. Also, y
Hi Matthias,
This doeesn't look possible now. It may be worth filing an improvement
jira for.
But I'm trying to understand what you're trying to do a little better. So
you intentionally have each thread create a new unique pool when its
submits a job? So that pool will just get the default poo
@Crystal
You can use spark on yarn. Yarn have fair scheduler,modified yarn-site.xml.
发自我的 iPad
> 在 2014年8月11日，6:49，Matei Zaharia 写道：
>
> Hi Crystal,
>
> The fair scheduler is only for jobs running concurrently within the same
> SparkContext (i.e. within an application), not for separate appli
Hi Crystal,
The fair scheduler is only for jobs running concurrently within the same
SparkContext (i.e. within an application), not for separate applications on the
standalone cluster manager. It has no effect there. To run more of those
concurrently, you need to set a cap on how many cores the
