[
https://issues.apache.org/jira/browse/HIVE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080283#comment-14080283
]
Xuefu Zhang commented on HIVE-7526:
-----------------------------------
Chao, Based on our last conversation, I don't think your patch is final or
ready to be reviewed. Please continue working on your patch and update when you
think it's ready. Here is what I have emphasized:
1. Define a SparkShuffle interface that's similar to existing ShuffleTran.
2. Have two implementation of this interface: sortBy and groupBy.
3. For sortBy, use a local key clustering mechanism.
4. Have ReduceTran contain a reference to SparkShuffle and HiveReduceFunction
instance.
Let me know if you have additional questions.
> Research to use groupby transformation to replace Hive existing
> partitionByKey and SparkCollector combination
> -------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-7526
> URL: https://issues.apache.org/jira/browse/HIVE-7526
> Project: Hive
> Issue Type: Task
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Chao
> Attachments: HIVE-7526.2.patch, HIVE-7526.3.patch, HIVE-7526.patch
>
>
> Currently SparkClient shuffles data by calling paritionByKey(). This
> transformation outputs <key, value> tuples. However, Hive's ExecMapper
> expects <key, iterator<value>> tuples, and Spark's groupByKey() seems
> outputing this directly. Thus, using groupByKey, we may be able to avoid its
> own key clustering mechanism (in HiveReduceFunction). This research is to
> have a try.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
