----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65422/#review198543 -----------------------------------------------------------
data/conf/hive-site.xml Lines 338 (patched) <https://reviews.apache.org/r/65422/#comment278729> Is there a reason for this? AFAICT existing tests should not get affected by this, so this shouldn't impact tests, so we might as well turn it to true. data/conf/llap/hive-site.xml Lines 363 (patched) <https://reviews.apache.org/r/65422/#comment278730> Set to true ? ql/src/java/org/apache/hadoop/hive/ql/Driver.java Lines 652 (patched) <https://reviews.apache.org/r/65422/#comment278763> We want to compute operator signatures and store stats corresponding to it only if we decide to reexecute it. Currently, it seems it unconditionally does this. Also, at this point plan hasn't been executed, so there are no stats for it. So, is this only computing signatures? ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java Lines 1581 (patched) <https://reviews.apache.org/r/65422/#comment278737> Also add a comment that its currently used only when reoptimization is attempted to match operators on 2 trees. ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g Lines 306 (patched) <https://reviews.apache.org/r/65422/#comment278739> Add it in non reserved list in IdentifiersParser.g ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java Lines 36 (patched) <https://reviews.apache.org/r/65422/#comment278762> These are not just filters, better name it as operators? ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java Lines 111 (patched) <https://reviews.apache.org/r/65422/#comment278765> Can there be a semantics about whether subsequent plugin run depends on results on plugins before it? I am thinking of a scenario where ReOptimize plugin executes first but it fails to generate an alternate plan (either because it can't get stats for an operator it was looking for or can't match op trees or whatever). So, we will know before executing second time via Reoptimize plugin that it won't be successful. At that time we make Overlay plugin go with auto.convert.join=off. I think this will be useful, although can be a follow-up. - Ashutosh Chauhan On March 1, 2018, 3:35 p.m., Zoltan Haindrich wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65422/ > ----------------------------------------------------------- > > (Updated March 1, 2018, 3:35 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Bugs: HIVE-17626 > https://issues.apache.org/jira/browse/HIVE-17626 > > > Repository: hive-git > > > Description > ------- > > preview > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0880a960e6 > data/conf/hive-site.xml b56cbd2469 > data/conf/llap/hive-site.xml c4c299c5de > data/conf/perf-reg/spark/hive-site.xml 5ca660d4e7 > data/conf/perf-reg/tez/hive-site.xml 62ecb74f3f > data/conf/rlist/hive-site.xml 630e481edc > data/conf/spark/local/hive-site.xml 8ff6256638 > data/conf/spark/standalone/hive-site.xml 84851c769c > data/conf/spark/yarn-client/hive-site.xml 6c633623ab > data/conf/tez/hive-site.xml 236adc7087 > > druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidScanQueryRecordReader.java > cbeac2c00a > itests/src/test/resources/testconfiguration.properties 2776fe95f1 > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java > 6cd7a136ae > itests/util/src/test/java/org/apache/hadoop/hive/ql/TestQTestUtil.java > c01d87bf51 > ql/src/java/org/apache/hadoop/hive/ql/Context.java dba2dbb15b > ql/src/java/org/apache/hadoop/hive/ql/Driver.java 94999fed93 > ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 60e8de8fd4 > ql/src/java/org/apache/hadoop/hive/ql/HookRunner.java 2a32a51588 > ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 9f13fa8e88 > ql/src/java/org/apache/hadoop/hive/ql/cache/results/CacheUsage.java > 08b791ad42 > ql/src/java/org/apache/hadoop/hive/ql/cache/results/QueryResultsCache.java > 131127e50d > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 32fc257b03 > ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewDesc.java > 1e28ca843f > ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java > 2b345d6ec7 > ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 > ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java > 395a5f450f > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveInputCounters.java > 085d6a7d94 > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectSubCache.java > 0d31e6e422 > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java > 8dd7cfe58c > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java > 134fc0ff0b > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java > 1eb72ce4d9 > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java > 384bd74686 > ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveException.java > b75850760f > ql/src/java/org/apache/hadoop/hive/ql/optimizer/SharedWorkOptimizer.java > b0cf3bd94e > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOpMaterializationValidator.java > 8c1bcb3f62 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > 783a672c47 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpSignature.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignature.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/signature/OpTreeSignatureFactory.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkPartitionPruningSinkDesc.java > d1c53cf345 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java > 9a3f81c98f > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 78cbf25c43 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 733ec79ce1 > ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractOperatorDesc.java > 714cf3914b > ql/src/java/org/apache/hadoop/hive/ql/plan/AppMasterEventDesc.java > 7d5be6ba81 > ql/src/java/org/apache/hadoop/hive/ql/plan/CommonMergeJoinDesc.java > 7332693513 > ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPruningEventDesc.java > 5d3fdb8b63 > ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ce61fc5a2e > ql/src/java/org/apache/hadoop/hive/ql/plan/FilterDesc.java d59834ce08 > ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 86cc77d43b > ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java > 9c651ab3ab > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinCondDesc.java 6dcf05af28 > ql/src/java/org/apache/hadoop/hive/ql/plan/JoinDesc.java bd45c752e1 > ql/src/java/org/apache/hadoop/hive/ql/plan/LateralViewJoinDesc.java > 3837a49934 > ql/src/java/org/apache/hadoop/hive/ql/plan/LimitDesc.java ce53feae00 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java cf4ab606f2 > ql/src/java/org/apache/hadoop/hive/ql/plan/OperatorDesc.java 870b4d9a26 > ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java bf24ff8f09 > ql/src/java/org/apache/hadoop/hive/ql/plan/ScriptDesc.java 858de98229 > ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java e38e7e468f > ql/src/java/org/apache/hadoop/hive/ql/plan/Signature.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/SignatureUtils1.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 > ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 59968fa8ef > ql/src/java/org/apache/hadoop/hive/ql/plan/UDTFDesc.java cf8e6e547e > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/IReExecutionPlugin.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecDriver.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReExecutionOverlayPlugin.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/reexec/ReOptimizePlugin.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFEnforceNotNullConstraint.java > 6c469bf944 > > ql/src/test/org/apache/hadoop/hive/ql/exec/vector/TestVectorGroupByOperator.java > fa3f89fda1 > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOpSigFactory.java > PRE-CREATION > > ql/src/test/org/apache/hadoop/hive/ql/optimizer/signature/TestOperatorSignature.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestOperatorCmp.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java > PRE-CREATION > ql/src/test/org/apache/hive/testutils/HiveTestEnvSetup.java cc97317f55 > ql/src/test/org/apache/hive/testutils/TestHiveTestEnvSetup.java 91613667f8 > ql/src/test/org/apache/hive/testutils/TestX1.java PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure_stat_changes.q > PRE-CREATION > ql/src/test/results/clientpositive/llap/dp_counter_mm.q.out 869916078b > ql/src/test/results/clientpositive/llap/dp_counter_non_mm.q.out 9f6426ccd1 > ql/src/test/results/clientpositive/llap/orc_llap_counters.q.out aa2dcc7f68 > ql/src/test/results/clientpositive/llap/orc_llap_counters1.q.out eb54a81d1a > ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 590437c708 > ql/src/test/results/clientpositive/llap/orc_ppd_schema_evol_3a.q.out > c7c8993d0e > ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION > ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/tez_input_counters.q.out 71d14f4049 > ql/src/test/results/clientpositive/show_functions.q.out 4e8ceaf6af > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/hooks/URIResolverHook.java > d3be5dd0a2 > > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/messaging/CreateDatabaseMessage.java > 3d64c7341a > > > Diff: https://reviews.apache.org/r/65422/diff/6/ > > > Testing > ------- > > > Thanks, > > Zoltan Haindrich > >
