> On April 7, 2014, 6:03 p.m., John Pullokkaran wrote: > > I took a look at this change; my knowledge of hive code is rather limited. > > 1. Column Pruner doesn't cross Script operator boundary. Theoretically you > > could prune above and below the script op separately. > > 2. It seems column pruner assumes that parent of UDTF is always select; but > > we haven't formalized this assumption. Other processors should throw > > exception if it ever come across a child that is UDTF. Theoretically you > > can push down certain filters below builtin UDTF. We may not be doing that > > today. > > 3. In Select Pruner it seems like there is no difference between > > 'prunedCols' and 'columns'.
Thanks John.Here are responses to your points 1. Column Pruner doesn't cross Script operator boundary. The ColumnPrunerWalker explicitly stops at the SelectOp parent of a ScriptOp. This may have been ok when developed; as you point out now it makes sense to continue pruning on the SelectOp ancestors. Can you file a jira for this. 2. The check in ColumnPrunerSelectProc is needed for the LVJoin case, where for the UDTFOp you end up with a empty PrunedList. What I realized was that Navis's fix doesn't cover the LVJoin case. Yes this should be revisited. - Harish ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/20051/#review39706 ----------------------------------------------------------- On April 6, 2014, 1:33 a.m., Harish Butani wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/20051/ > ----------------------------------------------------------- > > (Updated April 6, 2014, 1:33 a.m.) > > > Review request for hive, Ashutosh Chauhan and Navis Ryu. > > > Bugs: HIVE-4904 > https://issues.apache.org/jira/browse/HIVE-4904 > > > Repository: hive-git > > > Description > ------- > > Currently, CP context cannot be propagated over RS except for JOIN/EXT. A > little more CP is possible. > > > Diffs > ----- > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPruner.java 58a9b59 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcCtx.java > db36151 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/ColumnPrunerProcFactory.java > 0690fb7 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 3f16dc2 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java > 94224b3 > ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 3b33dc2 > ql/src/test/queries/clientpositive/order_within_subquery.q PRE-CREATION > ql/src/test/results/clientpositive/annotate_stats_select.q.out 1e982e6 > ql/src/test/results/clientpositive/auto_join18.q.out b8677f4 > ql/src/test/results/clientpositive/auto_join27.q.out a576190 > ql/src/test/results/clientpositive/auto_join30.q.out 8709198 > ql/src/test/results/clientpositive/auto_join31.q.out 1936e45 > ql/src/test/results/clientpositive/auto_join32.q.out 05f53e6 > ql/src/test/results/clientpositive/auto_sortmerge_join_10.q.out 8882aac > ql/src/test/results/clientpositive/count.q.out eb048b6 > ql/src/test/results/clientpositive/distinct_stats.q.out f715ea3 > ql/src/test/results/clientpositive/groupby2_map.q.out 291f196 > ql/src/test/results/clientpositive/groupby2_map_skew.q.out d005b6c > ql/src/test/results/clientpositive/groupby3_map.q.out 1dfee08 > ql/src/test/results/clientpositive/groupby3_map_skew.q.out 7af59bc > ql/src/test/results/clientpositive/groupby_cube1.q.out 92d81f4 > ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out b405978 > ql/src/test/results/clientpositive/groupby_map_ppr.q.out 27eff75 > > ql/src/test/results/clientpositive/groupby_multi_insert_common_distinct.q.out > ad76252 > ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out > 51a70c4 > ql/src/test/results/clientpositive/groupby_position.q.out 727bccb > ql/src/test/results/clientpositive/groupby_rollup1.q.out 36bf966 > ql/src/test/results/clientpositive/groupby_sort_11.q.out 8ee7571 > ql/src/test/results/clientpositive/groupby_sort_8.q.out a27cfaa > ql/src/test/results/clientpositive/join18.q.out 7975c79 > ql/src/test/results/clientpositive/limit_pushdown.q.out 9c93ada > ql/src/test/results/clientpositive/limit_pushdown_negative.q.out 115b171 > ql/src/test/results/clientpositive/metadataonly1.q.out 917efdf > ql/src/test/results/clientpositive/multi_insert_gby2.q.out ab758cb > ql/src/test/results/clientpositive/multi_insert_gby3.q.out 23ccebb > ql/src/test/results/clientpositive/multi_insert_lateral_view.q.out 35e70b4 > ql/src/test/results/clientpositive/nullgroup.q.out 2ac7dea > ql/src/test/results/clientpositive/nullgroup2.q.out cf31dc1 > ql/src/test/results/clientpositive/nullgroup4.q.out feae138 > ql/src/test/results/clientpositive/nullgroup4_multi_distinct.q.out 2ee357f > ql/src/test/results/clientpositive/order_within_subquery.q.out PRE-CREATION > ql/src/test/results/clientpositive/reduce_deduplicate_extended.q.out > 9c6d14e > ql/src/test/results/clientpositive/udf_count.q.out fb45708 > ql/src/test/results/clientpositive/union11.q.out f226f35 > ql/src/test/results/clientpositive/union14.q.out a6d349b > ql/src/test/results/clientpositive/union15.q.out 88c9553 > ql/src/test/results/clientpositive/union16.q.out 2bd8d5e > ql/src/test/results/clientpositive/union2.q.out 0fac9d9 > ql/src/test/results/clientpositive/union25.q.out 1ebe682 > ql/src/test/results/clientpositive/union28.q.out 4252062 > ql/src/test/results/clientpositive/union3.q.out 7f1b7fb > ql/src/test/results/clientpositive/union30.q.out 194b3b8 > ql/src/test/results/clientpositive/union31.q.out 2f7031f > ql/src/test/results/clientpositive/union5.q.out 0087393 > ql/src/test/results/clientpositive/union7.q.out 3a2d88c > ql/src/test/results/clientpositive/union9.q.out c6cc511 > ql/src/test/results/clientpositive/union_view.q.out 4eaaeaa > ql/src/test/results/clientpositive/vectorization_limit.q.out 4ffcd03 > ql/src/test/results/compiler/plan/groupby2.q.xml 58623d9 > ql/src/test/results/compiler/plan/groupby3.q.xml 65b403b > > Diff: https://reviews.apache.org/r/20051/diff/ > > > Testing > ------- > > > Thanks, > > Harish Butani > >
