I've noticed that DataSet.sqlContext is public in Scala but the equivalent
(DataFrame._sc) in PySpark is named as if it should be treated as private.
Is this intentional? If so, what's the rationale? If not, then it feels
like a bug and DataFrame should have some form of public access back to th
I'm curious to see the feedback others will provide. My impression is the
only way to get Spark to give up resources while it is idle would be to use
the preemption feature of the scheduler you're using in YARN. When another
user comes along the scheduler would preempt one or more Spark executors
Oops - I meant while it is *busy* when I said while it is *idle*.
On Tue, Dec 15, 2015 at 11:35 AM Ben Roling wrote:
> I'm curious to see the feedback others will provide. My impression is the
> only way to get Spark to give up resources while it is idle would be to use
>
