We run large multi-tenant clusters with spark/hadoop workloads, and we use
'yarn's preemption'/'spark's dynamic allocation' to achieve multitenancy.
See following link on how to enable/configure preemption using fair
scheduler :
http://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/Fai
I'm curious to see the feedback others will provide. My impression is the
only way to get Spark to give up resources while it is idle would be to use
the preemption feature of the scheduler you're using in YARN. When another
user comes along the scheduler would preempt one or more Spark executors
Oops - I meant while it is *busy* when I said while it is *idle*.
On Tue, Dec 15, 2015 at 11:35 AM Ben Roling wrote:
> I'm curious to see the feedback others will provide. My impression is the
> only way to get Spark to give up resources while it is idle would be to use
> the preemption feature
Hello Spark experts,
We are currently evaluating Spark on our cluster that already supports MRv2
over YARN.
We have noticed a problem with running jobs concurrently, in particular
that a running Spark job will not release its resources until the job is
finished. Ideally, if two people run any co
