if i wanted to pimp DataFrame to add subtract and intersect myself with a
physical operator, without needing to modify spark directly, is that
currently possible/intended? or will i run into the private[spark] issue?
On Fri, Nov 27, 2015 at 7:36 PM, Reynold Xin wrote:
> We need to first implemen
We need to first implement subtract and intersect in Spark SQL natively first
(i.e. add physical operator for them rather than using RDD.subtract/intersect).
Then it should be pretty easy to do that, given it is just about injecting the
right exchange operators.
> On Nov 27, 2015, at 11:19 PM
Hi,
I have seen massive gains with the broadcast hint for joins with
DataFrames, and I was wondering if we have thought about allowing the
broadcast hint for the implementation of subtract and intersect.
Right now, when I try it, it says that there is no plan for the broadcast
hint.
Justin
