[
https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13533638#comment-13533638
]
Phabricator commented on HIVE-3562:
-----------------------------------
njain has commented on the revision "HIVE-3562 [jira] Some limit can be pushed
down to map stage".
The general direction looks OK
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java:79
TODO
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java:45
spelling: operator
ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java:79
Followup: add a new method in Operator.
ql/src/test/queries/clientpositive/limit_pushdown.q:26 Looks like this
optimization should also help if the limit is in a sub-query:
Can you add a test ?
something like:
select .. from
(select key, count(1) from src group by key order by key limit 2) subq
join
(select key, count(1) from src group by key order by key limit 2) subq2 ..
The optimization should be applied to both the sub-queries
REVISION DETAIL
https://reviews.facebook.net/D5967
BRANCH
DPAL-1910
To: JIRA, tarball, navis
Cc: njain
> Some limit can be pushed down to map stage
> ------------------------------------------
>
> Key: HIVE-3562
> URL: https://issues.apache.org/jira/browse/HIVE-3562
> Project: Hive
> Issue Type: Bug
> Reporter: Navis
> Assignee: Navis
> Priority: Trivial
> Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch
>
>
> Queries with limit clause (with reasonable number), for example
> {noformat}
> select * from src order by key limit 10;
> {noformat}
> makes operator tree,
> TS-SEL-RS-EXT-LIMIT-FS
> But LIMIT can be partially calculated in RS, reducing size of shuffling.
> TS-SEL-RS(TOP-N)-EXT-LIMIT-FS
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
