[
https://issues.apache.org/jira/browse/HIVE-17486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16285550#comment-16285550
]
Xuefu Zhang commented on HIVE-17486:
------------------------------------
[~kellyzly], Thanks for working on this. I'm not sure if we should just look at
TS to determine whether to generate M-M-R. It seems that we can do so whenever
an TS is connected to multiple RSs. The split point should happen at the fork.
I'm not sure what's the best way to apply the optimization rules, but if you
look at SparkProcessAnalyzeTable, it has an if statement to check if it's an
analyze table command. If not, it doesn't do anything. Thus, you can have a
super rule that covers both analyze table and the new rule you're adding.
> Enable SharedWorkOptimizer in tez on HOS
> ----------------------------------------
>
> Key: HIVE-17486
> URL: https://issues.apache.org/jira/browse/HIVE-17486
> Project: Hive
> Issue Type: Bug
> Reporter: liyunzhang
> Assignee: liyunzhang
> Attachments: HIVE-17486.1.patch, explain.28.share.false,
> explain.28.share.true, scanshare.after.svg, scanshare.before.svg
>
>
> in HIVE-16602, Implement shared scans with Tez.
> Given a query plan, the goal is to identify scans on input tables that can be
> merged so the data is read only once. Optimization will be carried out at the
> physical level. In Hive on Spark, it caches the result of spark work if the
> spark work is used by more than 1 child spark work. After sharedWorkOptimizer
> is enabled in physical plan in HoS, the identical table scans are merged to 1
> table scan. This result of table scan will be used by more 1 child spark
> work. Thus we need not do the same computation because of cache mechanism.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
