[
https://issues.apache.org/jira/browse/HIVE-15796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860365#comment-15860365
]
Chao Sun commented on HIVE-15796:
---------------------------------
Thanks for looking at this, [~xuefuz]!
bq. 1. Since we are changing the behavior, I think the default value should
retain the old behavior.
Yes, I also plan to retain the old behavior. I set the the default to new
behavior so that tests can be triggered.
bq. 2. In setSparkReduceParallelism, I think it's probably cleaner to have a
separate method for the new implementation rather than using several if/else
blocks.
Will do this in the next patch.
bq. 3. Looking at the test out diff, I'm not sure if the change of number of
reducers are due to the new implementation.
Yes it is because of the new implementation. The old implementation uses Spark
# of cores (which I think is 2 in the test environment) to set the # of
reducers, while the new implementation inherits the # from parents, thus the
difference.
bq. 4. Wondering if using the default graph walker is absolutely needed or just
for some benefits of this new implementation.
The PreOrder walker is needed because for a particular RS we want to process
all of its parent RSs first. However, changing to PreOrder walker means some of
the plan outputs also get changed. Another way is to apply PreOrder walker just
for the {{SetSparkReducerParallelism}}, but this means we need one extra pass
on the OP tree.
> HoS: poor reducer parallelism when operator stats are not accurate
> ------------------------------------------------------------------
>
> Key: HIVE-15796
> URL: https://issues.apache.org/jira/browse/HIVE-15796
> Project: Hive
> Issue Type: Improvement
> Components: Statistics
> Affects Versions: 2.2.0
> Reporter: Chao Sun
> Assignee: Chao Sun
> Attachments: HIVE-15796.1.patch, HIVE-15796.2.patch,
> HIVE-15796.wip.1.patch, HIVE-15796.wip.2.patch, HIVE-15796.wip.patch
>
>
> In HoS we use currently use operator stats to determine reducer parallelism.
> However, it is often the case that operator stats are not accurate,
> especially if column stats are not available. This sometimes will generate
> extremely poor reducer parallelism, and cause HoS query to run forever.
> This JIRA tries to offer an alternative way to compute reducer parallelism,
> similar to how MR does. Here's the approach we are suggesting:
> 1. when computing the parallelism for a MapWork, use stats associated with
> the TableScan operator;
> 2. when computing the parallelism for a ReduceWork, use the *maximum*
> parallelism from all its parents.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
