> > 2. Does 1.3.2 or 1.4 have any enhancements that can help? I tried to use > 1.3.1 but SPARK-6967 prohibits me from doing so. Now that 1.4 is > available, would any of the JOIN enhancements help this situation? >
I would try Spark 1.4 after running "SET spark.sql.planner.sortMergeJoin=true". Please report back if this works for you.
