----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16440/ -----------------------------------------------------------
(Updated Dec. 27, 2013, 6:38 a.m.) Review request for hive. Changes ------- Updated patch contains the latest changes (rebase + class path stuff) Repository: hive Description ------- I think the Tez branch is at a point where we can consider merging it back into trunk after review. Tez itself has had its first release, most hive features are available on Tez and the test coverage is decent. There are a few known limitations, all of which can be handled in trunk as far as I can tell (i.e.: None of them are large disruptive changes that still require a branch.) Limitations: Union all is not yet supported on Tez SMB is not yet supported on Tez Bucketed map-join is executed as broadcast join (bucketing is ignored) Since the user is free to toggle hive.optimize.tez, it's obviously possible to just run these on MR. I am hoping to follow the approach that was taken with vectorization and shoot for a merge instead of single commit. This would retain history of the branch. Also in vectorization we required at least three +1s before merge, I'm hoping to go with that as well. I will add a combined patch to this ticket for review purposes (not for commit). I'll also attach instructions to run on a cluster if anyone wants to try. Diffs (updated) ----- trunk/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java 1553631 trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1553631 trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 1553631 trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1553631 trunk/conf/hive-default.xml.template 1553631 trunk/data/conf/tez/hive-site.xml UNKNOWN trunk/hbase-handler/src/test/templates/TestHBaseCliDriver.vm 1553631 trunk/hbase-handler/src/test/templates/TestHBaseNegativeCliDriver.vm 1553631 trunk/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestLocationQueries.java 1553631 trunk/itests/qtest/pom.xml 1553631 trunk/itests/util/src/main/java/org/apache/hadoop/hive/hbase/HBaseQTestUtil.java 1553631 trunk/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 1553631 trunk/pom.xml 1553631 trunk/ql/pom.xml 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/HashTableLoaderFactory.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredContext.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MuxOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheFactory.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorUtils.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecReducer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HadoopJobExecHelper.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapTezProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceTezProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezContext.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/tools/InputMerger.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorReduceSinkOperator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/IOContext.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSplit.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcInputFormat.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/CompositeProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/lib/DefaultGraphWalker.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/log/PerfLogger.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/AbstractSMBJoinProc.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SetReducerParallelism.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/SkewJoinOptimizer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationOptimizer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FileSinkProcessor.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezProcContext.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWork.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezWorkWalker.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/OptimizeTezProcContext.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TableAccessAnalyzer.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompilerFactory.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TezWalker.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceWork.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/TezWork.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsAggregator.java 1553631 trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/jdbc/JDBCStatsAggregator.java 1553631 trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/TestOperators.java 1553631 trunk/ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezTask.java PRE-CREATION trunk/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 1553631 trunk/ql/src/test/org/apache/hadoop/hive/ql/parse/TestGenTezWork.java PRE-CREATION trunk/ql/src/test/org/apache/hadoop/hive/ql/plan/TestTezWork.java PRE-CREATION trunk/ql/src/test/org/apache/hadoop/hive/ql/session/TestSessionState.java 1553631 trunk/ql/src/test/queries/clientpositive/mrr.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/tez_dml.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/tez_insert_overwrite_local_directory_1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/tez_join_tests.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/tez_joins_explain.q PRE-CREATION trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 1553631 trunk/ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 1553631 trunk/ql/src/test/results/clientpositive/join32.q.out 1553631 trunk/ql/src/test/results/clientpositive/join32_lessSize.q.out 1553631 trunk/ql/src/test/results/clientpositive/join33.q.out 1553631 trunk/ql/src/test/results/clientpositive/join34.q.out 1553631 trunk/ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out 1553631 trunk/ql/src/test/results/clientpositive/tez/auto_join0.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/auto_join1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/bucket2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/bucket3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/bucket4.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/count.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/create_merge_compressed.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/cross_join.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/ctas.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/custom_input_output_format.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/enforce_order.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/fileformat_mix.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/filter_join_breaktask2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/groupby1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/groupby2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/groupby3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/having.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/insert1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/insert_into1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/insert_into2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/join0.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/join1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/leftsemijoin.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/load_dyn_part2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/load_dyn_part3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/mapreduce1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/mapreduce2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/merge1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/merge2.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/mrr.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/ptf.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/sample1.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/subquery_exists.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/subquery_in.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/tez_dml.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/tez_insert_overwrite_local_directory_1.q.out UNKNOWN trunk/ql/src/test/results/clientpositive/tez/tez_join_tests.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/tez/vectorization_15.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/vector_left_outer_join.q.out 1553631 trunk/ql/src/test/results/clientpositive/vectorized_mapjoin.q.out 1553631 trunk/ql/src/test/results/compiler/plan/case_sensitivity.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/cast1.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/groupby1.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/groupby2.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/groupby3.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/groupby4.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/groupby5.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/groupby6.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input1.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input2.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input20.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input3.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input4.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input5.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input6.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input7.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input8.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input9.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input_part1.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input_testsequencefile.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input_testxpath.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/input_testxpath2.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/join1.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/join2.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/join3.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/join4.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/join5.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/join6.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/join7.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/join8.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/sample2.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/sample3.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/sample4.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/sample5.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/sample6.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/sample7.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/subq.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/udf1.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/udf4.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/udf6.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/udf_case.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/udf_when.q.xml 1553631 trunk/ql/src/test/results/compiler/plan/union.q.xml 1553631 trunk/ql/src/test/templates/TestCliDriver.vm 1553631 trunk/ql/src/test/templates/TestNegativeCliDriver.vm 1553631 trunk/ql/src/test/templates/TestParse.vm 1553631 trunk/ql/src/test/templates/TestParseNegative.vm 1553631 trunk/shims/0.23/pom.xml 1553631 trunk/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 1553631 Diff: https://reviews.apache.org/r/16440/diff/ Testing ------- Testing has been done as a combination of unit tests and q file tests. Unit tests have been created for new classes where possible, new q file tests cover the delta. I've also gone through all .q file tests and picked the relevant ones to run on MiniTezCliDriver. Thanks, Gunther Hagleitner
