Good to know this will be in next release. Thanks.
On Wed, Jul 1, 2015 at 3:13 PM, Michael Armbrust
wrote:
> We don't know that the table is small unless you cache it. In Spark 1.5
> you'll be able to give us a hint though (
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/
We don't know that the table is small unless you cache it. In Spark 1.5
you'll be able to give us a hint though (
https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L581
)
On Wed, Jul 1, 2015 at 8:30 AM, Srikanth wrote:
> Hello,
>
>
>
> I ha
Hello,
I have a straight forward use case of joining a large table with a smaller
table. The small table is within the limit I set for
spark.sql.autoBroadcastJoinThreshold.
I notice that ShuffledHashJoin is used to perform the join.
BroadcastHashJoin was used only when I pre-fetched and cached
