> On Aug. 29, 2014, 5:30 p.m., Brock Noland wrote: > > Hi Na, > > > > Thank you very much for the patch! I have one high level question: > > > > It appears we created the union_remove_spark* files because we wanted to > > add an additional property to the union_remove .q file? Meaning what is the > > delta beween union_remove_spark_1.q and union_remove_? > > > > Cheers! > > Na Yang wrote: > Hi Brock, > > That is correct. the union_remove_spark* files include an extra config > property hive.merge.sparkfile comparing to the corresponding union_remove_* > files. Except that extra config property, all other queries in the > union_remove_spark* file are same as the queries in the union_remove_* file. > > The hive.merge.sparkfile value is set according to the hive.merge.mapfile > and hive.merge.mapredfile properity values in the orginal union_remove_* > file. Regarding to the test result, we expect to see the same data are > returned from the union_remove_spark* queries and the corresponding > union_remove_* queries. > > Thanks, > Na > > Brock Noland wrote: > Hi, > > Thank you very much for the information! I think instead of adding the > new union_remove_spark tests we should just add the hive.merge.sparkfile > property to the union_remove q files. The extra property won't impact the > existng tests other than an extra line of outpit. > > If instead we'd like to keep the hive_remove_spark* properties then we'd > need to add a check to QTestUtil that does not run spark files for MR: > > > https://github.com/apache/hive/blob/trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java#L431 > > as the tests are currently running for both spark and MR. As such, I > think the first solution (just add the property to the existing tests) makes > sense. > > Thoughts? > > Na Yang wrote: > Hi Brock, > > Thank you for your suggestion. I also prefer the first solution. Let me > modify the existing union_remove q files and re-genenrate the .q.out files > for both MR and Spark. > > Thanks, > Na
Awesome, thanks!! - Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25176/#review51889 ----------------------------------------------------------- On Aug. 29, 2014, 6:44 a.m., Na Yang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25176/ > ----------------------------------------------------------- > > (Updated Aug. 29, 2014, 6:44 a.m.) > > > Review request for hive, Brock Noland, Szehon Ho, and Xuefu Zhang. > > > Bugs: HIVE-7870 > https://issues.apache.org/jira/browse/HIVE-7870 > > > Repository: hive-git > > > Description > ------- > > HIVE-7870: Insert overwrite table query does not generate correct task plan > [Spark Branch] > > The cause of this problem is during spark/tez task generation, the union file > sink operator are cloned to two new filesink operator. The linkedfilesinkdesc > info for those new filesink operators are missing. In addition, the two new > filesink operators also need to be linked together. > > > Diffs > ----- > > itests/src/test/resources/testconfiguration.properties 6393671 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 9c808d4 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java > 5ddc16d > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java > 379a39c > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java > 76fc290 > ql/src/test/queries/clientpositive/union_remove_spark_1.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_10.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_11.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_15.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_16.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_17.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_18.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_19.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_2.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_20.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_21.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_24.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_25.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_3.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_4.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_5.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_6.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_7.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_8.q PRE-CREATION > ql/src/test/queries/clientpositive/union_remove_spark_9.q PRE-CREATION > ql/src/test/results/clientpositive/spark/sample8.q.out c7e333b > ql/src/test/results/clientpositive/spark/union10.q.out 20c681e > ql/src/test/results/clientpositive/spark/union18.q.out 3f37a0a > ql/src/test/results/clientpositive/spark/union19.q.out 6922fcd > ql/src/test/results/clientpositive/spark/union28.q.out 8bd5218 > ql/src/test/results/clientpositive/spark/union29.q.out b9546ef > ql/src/test/results/clientpositive/spark/union3.q.out 3ae6536 > ql/src/test/results/clientpositive/spark/union30.q.out 12717a1 > ql/src/test/results/clientpositive/spark/union33.q.out b89757f > ql/src/test/results/clientpositive/spark/union4.q.out 6341cd9 > ql/src/test/results/clientpositive/spark/union6.q.out 263d9f4 > ql/src/test/results/clientpositive/spark/union_remove_spark_1.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_10.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_11.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_15.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_16.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_17.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_18.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_19.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_2.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_20.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_21.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_24.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_25.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_3.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_4.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_5.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_6.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_7.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_8.q.out > PRE-CREATION > ql/src/test/results/clientpositive/spark/union_remove_spark_9.q.out > PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_1.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_10.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_11.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_15.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_16.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_17.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_18.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_19.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_2.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_20.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_21.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_24.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_25.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_3.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_4.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_5.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_6.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_7.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_8.q.out PRE-CREATION > ql/src/test/results/clientpositive/union_remove_spark_9.q.out PRE-CREATION > > Diff: https://reviews.apache.org/r/25176/diff/ > > > Testing > ------- > > > Thanks, > > Na Yang > >
