Hi all,
how can I change (globally, for Zeppelin) the default scheduler pool which
SparkInterpreter submits jobs to. Currently all jobs go into the pool
'default' but I want them to go into the pool 'fair'.
We use "Per Note" and "scoped" processes for best resource sharing.
"spark.scheduler.pool"="fair" in Interpreter Settings does not work, should
it?
What works is
sc.setLocalProperty("spark.scheduler.pool","fair")
but it's required in every *note* (not just notebook) since it's on thread
level.
Is there a possibility to globally/per notebook set the 'fair' pool as the
default pool?
Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'.
Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.
This is awesome and together with dynamicAllocation allows for super
flexible usage for multiple users but above behavior is a bit complicated.
Thanks,
Fabian
