[
https://issues.apache.org/jira/browse/HIVE-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-8202:
----------------------------
Attachment: HIVE-8202.1-spark.patch
This passes all the auto_convert_join.* tests. It is as I outlined in
design-doc. Most optimization happens in logical layer even before
SparkCompiler, all we do is attach the localWork to the MapWork with the right
informations.
I struggled for awhile about the bucket names not being same, for same data in
different tables. It seems just changing file-sink operations to use MR-style
shuffle (w/HashPartitioner) fixes it, as it was using RangePartitioner before.
However, this affects bunch of query plans so attaching first patch for a spin.
Also, I have a test about inserting to two tables via MR and spark, and doing
SMB join on them to verify bucket-name is consistent for same data. It seems
to work. But I probably need HIVE-7801 to checkin that test, as it will make
sense to run it only on Spark, maybe it can come later.
I even tried the test
> Support SMB Join for Hive on Spark [Spark Branch]
> -------------------------------------------------
>
> Key: HIVE-8202
> URL: https://issues.apache.org/jira/browse/HIVE-8202
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Szehon Ho
> Attachments: HIVE-8202.1-spark.patch, Hive on Spark SMB Join.docx,
> Hive on Spark SMB Join.pdf
>
>
> SMB joins are used wherever the tables are sorted and bucketed. It's a
> reduce-side join. The join boils down to just merging the already sorted
> tables, allowing this operation to be faster than an ordinary map-join.
> However, if the tables are partitioned, there could be a slow down as each
> mapper would need to get a very small chunk of a partition which has a single
> key. Thus, in some scenarios it's beneficial to convert SMB join to SMB map
> join as well.
> The task is to research and support the conversion from regular SMB join to
> SMB map join for Spark execution engine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
