> On 六月 19, 2015, 1:47 p.m., Xuefu Zhang wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java, > > line 98 > > <https://reviews.apache.org/r/34757/diff/3/?file=988066#file988066line98> > > > > I think the recursion should go on even if there is only one child for > > a given work. For examle, if we have: > > > > w1 > > | > > w2 > > | > > w3 > > / \ > > w4 w5 > > > > Even if each of w1 and w2 has only one child, it's still possible that > > we can combine w4 and w5.
created HIVE-11082 to track this. > On 六月 19, 2015, 1:47 p.m., Xuefu Zhang wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java, > > line 207 > > <https://reviews.apache.org/r/34757/diff/3/?file=988066#file988066line207> > > > > Could you explain the reason here? add comments in latest patch. While combine multi equivalent works into single one, we need to update all the references to the replaced works. leave works output should be read by further SparkWork/FetchWork, we does not able to update work reference across SparkWork, so combine leave works may lead to error. - chengxiang ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34757/#review88537 ----------------------------------------------------------- On 六月 23, 2015, 7:24 a.m., chengxiang li wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/34757/ > ----------------------------------------------------------- > > (Updated 六月 23, 2015, 7:24 a.m.) > > > Review request for hive and Xuefu Zhang. > > > Bugs: HIVE-10844 > https://issues.apache.org/jira/browse/HIVE-10844 > > > Repository: hive-git > > > Description > ------- > > Some Hive queries(like TPCDS Q39) may share the same subquery, which > translated into sperate, but equivalent Works in SparkWork, combining these > equivalent Works into a single one would help to benifit from following > dynamic RDD caching optimization. > > > Diffs > ----- > > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/OperatorComparatorFactory.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/CombineEquivalentWorkResolver.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java > 19aae70 > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java b307b16 > ql/src/test/results/clientpositive/spark/auto_join30.q.out 7b5c5e7 > ql/src/test/results/clientpositive/spark/auto_smb_mapjoin_14.q.out 8a43d78 > ql/src/test/results/clientpositive/spark/groupby10.q.out dd9d9fe > ql/src/test/results/clientpositive/spark/groupby7_map.q.out abd6459 > ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out 5e69b31 > ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 3418b99 > > ql/src/test/results/clientpositive/spark/groupby7_noskew_multi_single_reducer.q.out > 2cb126d > ql/src/test/results/clientpositive/spark/groupby8.q.out c249b61 > ql/src/test/results/clientpositive/spark/groupby8_map_skew.q.out 2fb1d73 > ql/src/test/results/clientpositive/spark/insert_into3.q.out 7df5ba8 > ql/src/test/results/clientpositive/spark/join22.q.out b1e5b67 > ql/src/test/results/clientpositive/spark/skewjoinopt11.q.out 8a278ef > ql/src/test/results/clientpositive/spark/union10.q.out 5e8fe38 > ql/src/test/results/clientpositive/spark/union11.q.out 20c27c7 > ql/src/test/results/clientpositive/spark/union20.q.out 6f0dca6 > ql/src/test/results/clientpositive/spark/union28.q.out 98582df > ql/src/test/results/clientpositive/spark/union3.q.out 834b6d4 > ql/src/test/results/clientpositive/spark/union30.q.out 3409623 > ql/src/test/results/clientpositive/spark/union4.q.out c121ef0 > ql/src/test/results/clientpositive/spark/union5.q.out afee988 > ql/src/test/results/clientpositive/spark/union_remove_1.q.out ba0e293 > ql/src/test/results/clientpositive/spark/union_remove_15.q.out 26cfbab > ql/src/test/results/clientpositive/spark/union_remove_16.q.out 7a7aaf2 > ql/src/test/results/clientpositive/spark/union_remove_18.q.out a5e15c5 > ql/src/test/results/clientpositive/spark/union_remove_19.q.out ad44400 > ql/src/test/results/clientpositive/spark/union_remove_20.q.out 1d67177 > ql/src/test/results/clientpositive/spark/union_remove_21.q.out 9f5b070 > ql/src/test/results/clientpositive/spark/union_remove_22.q.out 2e01432 > ql/src/test/results/clientpositive/spark/union_remove_24.q.out 2659798 > ql/src/test/results/clientpositive/spark/union_remove_25.q.out 0a94684 > ql/src/test/results/clientpositive/spark/union_remove_4.q.out 6c3d596 > ql/src/test/results/clientpositive/spark/union_remove_6.q.out cd36189 > ql/src/test/results/clientpositive/spark/union_remove_6_subq.q.out c981ae4 > ql/src/test/results/clientpositive/spark/union_remove_7.q.out 084fbd6 > ql/src/test/results/clientpositive/spark/union_top_level.q.out dede1ef > > Diff: https://reviews.apache.org/r/34757/diff/ > > > Testing > ------- > > > Thanks, > > chengxiang li > >
