> On Oct. 19, 2014, 12:15 a.m., Xuefu Zhang wrote: > > ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q, line 1 > > <https://reviews.apache.org/r/26706/diff/4/?file=724864#file724864line1> > > > > Could we put this test as spark only, as splitting doesn't apply mr or > > tez? I think we have a dir for spark only tests. > > Chao Sun wrote: > I also wanted to make this as a spark-only test. > But the feature hasn't been implemented yet (I think Szehon is working on > it). > I made the file name to start with "spark_" so in future we can move it > to spark-only test directory. > But currently, there's no test dir for spark, only result dir. > > Xuefu Zhang wrote: > In the case, let rename the test to have a generic name. it's a valid > test case for MR also, but also a special case for Spark.
OK, thanks. I've updated the patch accordingly. - Chao ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26706/#review57286 ----------------------------------------------------------- On Oct. 19, 2014, 12:46 a.m., Chao Sun wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/26706/ > ----------------------------------------------------------- > > (Updated Oct. 19, 2014, 12:46 a.m.) > > > Review request for hive and Xuefu Zhang. > > > Bugs: HIVE-8436 > https://issues.apache.org/jira/browse/HIVE-8436 > > > Repository: hive-git > > > Description > ------- > > Based on the design doc, we need to split the operator tree of a work in > SparkWork if the work is connected to multiple child works. The way splitting > the operator tree is performed by cloning the original work and removing > unwanted branches in the operator tree. Please refer to the design doc for > details. > This process should be done right before we generate SparkPlan. We should > have a utility method that takes the orignal SparkWork and return a modified > SparkWork. > This process should also keep the information about the original work and its > clones. Such information will be needed during SparkPlan generation > (HIVE-8437). > > > Diffs > ----- > > itests/src/test/resources/testconfiguration.properties 558dd02 > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveBaseFunctionResultList.java > c956101 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java > 5153885 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java > 126cb9f > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java > 3773dcb > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java > d7744e9 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java > 280edde > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java > 644c681 > > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java > 1d01040 > > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java > 93940bc > > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java > 20eb344 > > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java > a62643a > ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1 > ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q > PRE-CREATION > ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81 > ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985 > ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b > ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa > ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out > 399fe41 > ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807 > ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412 > ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e > ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out > a43921e > ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44 > ql/src/test/results/clientpositive/spark/input13.q.out 260a65a > ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484 > ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d > ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb > ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b > ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc > ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d > ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f > ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893 > ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out > b07c582 > > ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out > fd477ca > ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out > 44991e3 > ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06 > ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d > > ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331 > ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b > ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543 > ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010 > ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b > ql/src/test/results/clientpositive/spark_multi_insert_split_work.q.out > PRE-CREATION > > Diff: https://reviews.apache.org/r/26706/diff/ > > > Testing > ------- > > All multi-insertion related results are regenerated, and manually checked > against the old results. > Also I created a new test "spark_multi_insert_spill_work.q" to check > splitting won't generate duplicate FSs. > > > Thanks, > > Chao Sun > >
