----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16440/#review31424 -----------------------------------------------------------
Ship it! Ship It! - Thejas Nair On Jan. 9, 2014, 1:52 a.m., Gunther Hagleitner wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/16440/ > ----------------------------------------------------------- > > (Updated Jan. 9, 2014, 1:52 a.m.) > > > Review request for hive. > > > Repository: hive > > > Description > ------- > > I think the Tez branch is at a point where we can consider merging it back > into trunk after review. > Tez itself has had its first release, most hive features are available on Tez > and the test coverage is decent. There are a few known limitations, all of > which can be handled in trunk as far as I can tell (i.e.: None of them are > large disruptive changes that still require a branch.) > Limitations: > Union all is not yet supported on Tez > SMB is not yet supported on Tez > Bucketed map-join is executed as broadcast join (bucketing is ignored) > Since the user is free to toggle hive.optimize.tez, it's obviously possible > to just run these on MR. > I am hoping to follow the approach that was taken with vectorization and > shoot for a merge instead of single commit. This would retain history of the > branch. Also in vectorization we required at least three +1s before merge, > I'm hoping to go with that as well. > I will add a combined patch to this ticket for review purposes (not for > commit). I'll also attach instructions to run on a cluster if anyone wants to > try. > > > Diffs > ----- > > trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1556697 > trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java > 1556697 > trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1556697 > trunk/conf/hive-default.xml.template 1556697 > trunk/data/conf/tez/hive-site.xml UNKNOWN > trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1556697 > trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm > 1556697 > trunk/hcatalog/webhcat/svr/src/test/data/status/hive/stderr 1556697 > > trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java > 1556697 > trunk/itests/qtest/pom.xml 1556697 > > trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java > 1556697 > trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > 1556697 > trunk/pom.xml 1556697 > trunk/ql/pom.xml 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java > 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java > 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1556697 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java > 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java > PRE-CREATION > > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java > PRE-CREATION > > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java > PRE-CREATION > > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1556697 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java > 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java > 1556697 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java > PRE-CREATION > > trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java > PRE-CREATION > > trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java > PRE-CREATION > > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java > 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java > 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java > PRE-CREATION > > trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java > 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1556697 > trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java > 1556697 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java > 1556697 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java > 1556697 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java > 1556697 > trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java > PRE-CREATION > trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java > PRE-CREATION > trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java > PRE-CREATION > trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java > 1556697 > trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION > trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION > > trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q > PRE-CREATION > trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION > trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION > > trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out > PRE-CREATION > > trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION > > trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out > UNKNOWN > trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out > PRE-CREATION > trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out > PRE-CREATION > trunk/ql/src/test/templates/TestCliDriver.vm 1556697 > trunk/ql/src/test/templates/TestCompareCliDriver.vm 1556697 > trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1556697 > trunk/ql/src/test/templates/TestParse.vm 1556697 > trunk/ql/src/test/templates/TestParseNegative.vm 1556697 > trunk/shims/0.23/pom.xml 1556697 > > trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java > 1556697 > > Diff: https://reviews.apache.org/r/16440/diff/ > > > Testing > ------- > > Testing has been done as a combination of unit tests and q file tests. Unit > tests have been created for new classes where possible, new q file tests > cover the delta. I've also gone through all .q file tests and picked the > relevant ones to run on MiniTezCliDriver. > > > Thanks, > > Gunther Hagleitner > >
