[
https://issues.apache.org/jira/browse/HIVE-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15397041#comment-15397041
]
Nemon Lou commented on HIVE-14353:
----------------------------------
A preliminary analysis:
Hive has a built in column pruner, and column pruning has been pushed down to
InputFormat layer.
CBO adds an projection above table scan,which is very costly especially when
doing projection before join.
Join can filter out a lot of rows in most cases of TPCDS.
> Performance degradation after Projection Pruning in CBO
> --------------------------------------------------------
>
> Key: HIVE-14353
> URL: https://issues.apache.org/jira/browse/HIVE-14353
> Project: Hive
> Issue Type: Bug
> Components: CBO, Logical Optimizer
> Affects Versions: 1.2.1
> Reporter: Nemon Lou
> Attachments: q46_cbo_no_projection_prune_explain.txt,
> q46_cbo_projection_prune_explain.rar
>
>
> TPC-DS with factor 1024.
> Hive on Spark.
> With and without projection prunning,time spent are quite different.
> The way to disable projection prunning : disable HiveRelFieldTrimmer in code
> and compile a new jar.
> ||queries||CBO_no_projection_prune||CBO||
> |q27| 160|251 |
> |q7 | 200|312 |
> |q88| 701|1092|
> |q68| 234|345 |
> |q39| 53|78 |
> |q73| 160|228 |
> |q31| 463|659 |
> |q79| 242|343 |
> |q46| 256|363 |
> |q60| 271|382 |
> |q66| 198|278 |
> |q34| 155|217 |
> |q19| 184|256 |
> |q26| 154|214 |
> |q56| 262|364 |
> |q75| 942|1303|
> |q71| 288|388 |
> |q25| 329|442 |
> |q52| 142|190 |
> |q42| 142|189 |
> |q3 | 139|185 |
> |q98| 153|203 |
> |q89| 187|248 |
> |q58| 264|340 |
> |q43| 127|162 |
> |q32| 174|221 |
> |q96| 156|197 |
> |q70| 320|404 |
> |q29| 499|629 |
> |q18| 266|329 |
> |q21| 76 |92 |
> |q90| 139|165 |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
