----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30388/#review70259 -----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java <https://reviews.apache.org/r/30388/#comment115338> static method? ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java <https://reviews.apache.org/r/30388/#comment115339> static method? ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java <https://reviews.apache.org/r/30388/#comment115340> These two lines should be probably put in the for loop below. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java <https://reviews.apache.org/r/30388/#comment115591> Please refer to my comments on previous versions for this section of code. ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java <https://reviews.apache.org/r/30388/#comment115589> Maybe name this method generateSparkTask? - Xuefu Zhang On Jan. 29, 2015, 10:40 p.m., Chao Sun wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30388/ > ----------------------------------------------------------- > > (Updated Jan. 29, 2015, 10:40 p.m.) > > > Review request for hive and Xuefu Zhang. > > > Bugs: HIVE-9103 > https://issues.apache.org/jira/browse/HIVE-9103 > > > Repository: hive-git > > > Description > ------- > > This patch adds backup task to map join task. The backup task, which uses > common join, will be triggered > in case the mapjoin task failed. > > Note that, no matter how many map joins there are in the SparkTask, we will > only generate one backup task. > This means that if the original task failed at the very last map join, the > whole task will be re-executed. > > The handling of backup task is a little bit different from what MR does, > mostly because we convert JOIN to > MAPJOIN during the operator plan optimization phase, at which time no > task/work exist yet. In the patch, we > cloned the whole operator tree before the JOIN operator is converted. The > operator tree will be processed > and generate a separate work tree for a separate backup SparkTask. > > > Diffs > ----- > > itests/src/test/resources/testconfiguration.properties f583aaf > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java > 69004dc > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/StageIDsRearranger.java > 79c3e02 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkJoinOptimizer.java > d57ceff > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java > 9ff47c7 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java > 6e0ac38 > ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java b838bff > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java > 773cfbd > > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/OptimizeSparkProcContext.java > f7586a4 > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java > 3a7477a > ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 0e85990 > ql/src/test/results/clientpositive/auto_join30.q.out cb3f777 > ql/src/test/results/clientpositive/auto_join31.q.out 46b47dd > ql/src/test/results/clientpositive/auto_join_stats.q.out 9100762 > ql/src/test/results/clientpositive/auto_join_without_localtask.q.out > fc44acf > ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 926c99a > ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out 597e423 > ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 4b392c7 > ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 9e33128 > ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out 694e72b > ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out c16aae0 > ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 91ad73e > ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 85b1458 > ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 9837c78 > ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out > 599b8b9 > ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out > fd99597 > ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out > 8130ab9 > ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out > 627aba0 > ql/src/test/results/clientpositive/correlationoptimizer5.q.out 6ba3462 > ql/src/test/results/clientpositive/cross_product_check_2.q.out 102eeb9 > ql/src/test/results/clientpositive/explain_rearrange.q.out c4a015e > ql/src/test/results/clientpositive/identity_project_remove_skip.q.out > e7bd0ed > ql/src/test/results/clientpositive/join29.q.out 6d650e3 > ql/src/test/results/clientpositive/join31.q.out 9248cd9 > ql/src/test/results/clientpositive/multiMapJoin1.q.out 0a64c79 > ql/src/test/results/clientpositive/multiMapJoin2.q.out b0bdfbd > ql/src/test/results/clientpositive/spark/auto_join0.q.out 7f8eb63 > ql/src/test/results/clientpositive/spark/auto_join1.q.out b640b9d > ql/src/test/results/clientpositive/spark/auto_join10.q.out f01765c > ql/src/test/results/clientpositive/spark/auto_join11.q.out 69c10e6 > ql/src/test/results/clientpositive/spark/auto_join12.q.out bc763ed > ql/src/test/results/clientpositive/spark/auto_join13.q.out 935ebf5 > ql/src/test/results/clientpositive/spark/auto_join14.q.out 830314e > ql/src/test/results/clientpositive/spark/auto_join15.q.out 780540b > ql/src/test/results/clientpositive/spark/auto_join16.q.out f705339 > ql/src/test/results/clientpositive/spark/auto_join17.q.out 3144db6 > ql/src/test/results/clientpositive/spark/auto_join19.q.out f2b0140 > ql/src/test/results/clientpositive/spark/auto_join2.q.out 2424cca > ql/src/test/results/clientpositive/spark/auto_join20.q.out 9258f3b > ql/src/test/results/clientpositive/spark/auto_join21.q.out aa8f6dd > ql/src/test/results/clientpositive/spark/auto_join22.q.out d49dda9 > ql/src/test/results/clientpositive/spark/auto_join23.q.out a179d87 > ql/src/test/results/clientpositive/spark/auto_join24.q.out cfb076e > ql/src/test/results/clientpositive/spark/auto_join25.q.out ab01b8a > ql/src/test/results/clientpositive/spark/auto_join26.q.out 58821e9 > ql/src/test/results/clientpositive/spark/auto_join28.q.out d30133b > ql/src/test/results/clientpositive/spark/auto_join29.q.out 780c6cb > ql/src/test/results/clientpositive/spark/auto_join3.q.out 54e24f3 > ql/src/test/results/clientpositive/spark/auto_join30.q.out 4c832e2 > ql/src/test/results/clientpositive/spark/auto_join31.q.out 5980814 > ql/src/test/results/clientpositive/spark/auto_join32.q.out 9629f53 > ql/src/test/results/clientpositive/spark/auto_join4.q.out 3366f75 > ql/src/test/results/clientpositive/spark/auto_join5.q.out b6d8798 > ql/src/test/results/clientpositive/spark/auto_join8.q.out 5b6cc80 > ql/src/test/results/clientpositive/spark/auto_join9.q.out 6daf348 > ql/src/test/results/clientpositive/spark/auto_join_filters.q.out 8934433 > ql/src/test/results/clientpositive/spark/auto_join_nulls.q.out 1f37c75 > ql/src/test/results/clientpositive/spark/auto_join_stats.q.out 1fa1a74 > ql/src/test/results/clientpositive/spark/auto_join_stats2.q.out c6473d3 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out > 3d465db > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_10.q.out > fe7b96d > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out > f4e889a > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out > c358721 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out > 7eadcd0 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_14.q.out > 984db20 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_15.q.out > 2acc323 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out > f05b0cc > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out > c7d23f8 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out > f5dc2f7 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out > 26e7957 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out > a5c0562 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out > ef13a40 > ql/src/test/results/clientpositive/spark/auto_sortmerge_join_9.q.out > a2b98fc > ql/src/test/results/clientpositive/spark/bucket_map_join_spark1.q.out > 6230bef > ql/src/test/results/clientpositive/spark/bucket_map_join_spark2.q.out > 1a33625 > ql/src/test/results/clientpositive/spark/bucket_map_join_spark3.q.out > fed923c > ql/src/test/results/clientpositive/spark/bucket_map_join_spark4.q.out > 8b5e8d4 > ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 1c81d1b > ql/src/test/results/clientpositive/spark/bucket_map_join_tez2.q.out 04a934f > ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_2.q.out > 365306e > ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_4.q.out > 3846de7 > ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_6.q.out > 5b559c4 > ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_7.q.out > cefc6aa > ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_8.q.out > ca44d7c > ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out > dda6c38 > ql/src/test/results/clientpositive/spark/identity_project_remove_skip.q.out > 7238009 > > ql/src/test/results/clientpositive/spark/infer_bucket_sort_convert_join.q.out > 3d4eb18 > ql/src/test/results/clientpositive/spark/join28.q.out f23f662 > ql/src/test/results/clientpositive/spark/join29.q.out 0b4284c > ql/src/test/results/clientpositive/spark/join31.q.out a52a8b6 > ql/src/test/results/clientpositive/spark/join32.q.out a9d50b4 > ql/src/test/results/clientpositive/spark/join32_lessSize.q.out dac9610 > ql/src/test/results/clientpositive/spark/join33.q.out a9d50b4 > ql/src/test/results/clientpositive/spark/join_reorder4.q.out 5cc30f7 > ql/src/test/results/clientpositive/spark/join_star.q.out 69c2fd7 > ql/src/test/results/clientpositive/spark/mapjoin_decimal.q.out b681e5f > ql/src/test/results/clientpositive/spark/mapjoin_filter_on_outerjoin.q.out > 0271f97 > ql/src/test/results/clientpositive/spark/mapjoin_hook.q.out 7aa8ce9 > ql/src/test/results/clientpositive/spark/mapjoin_mapjoin.q.out 65a7d06 > ql/src/test/results/clientpositive/spark/mapjoin_memcheck.q.out 14f316c > ql/src/test/results/clientpositive/spark/mapjoin_subquery.q.out 2d1e7a7 > ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out a757d0b > ql/src/test/results/clientpositive/spark/mapjoin_test_outer.q.out 7143348 > ql/src/test/results/clientpositive/spark/multi_join_union.q.out bda569d > ql/src/test/results/clientpositive/spark/parquet_join.q.out 390aeb1 > > ql/src/test/results/clientpositive/spark/reduce_deduplicate_exclude_join.q.out > 19ab4c8 > ql/src/test/results/clientpositive/spark/smb_mapjoin_17.q.out bd3a6a1 > ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out cb811ed > ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.java1.7.out > 92a8595 > ql/src/test/results/clientpositive/spark/vector_decimal_mapjoin.q.out > 5ec95c2 > ql/src/test/results/clientpositive/spark/vector_left_outer_join.q.out > ca8918a > ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out > 02c1fc6 > ql/src/test/results/clientpositive/spark/vectorized_mapjoin.q.out 237df98 > ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out > f8e8ba7 > ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 6f11b8c > > Diff: https://reviews.apache.org/r/30388/diff/ > > > Testing > ------- > > auto_join25.q > > > Thanks, > > Chao Sun > >
