> On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java > > Lines 131 (patched) > > <https://reviews.apache.org/r/65422/diff/1/?file=1950478#file1950478line131> > > > > This is hackish.. as pointed above it needs to happen via explain > > modifier.
I agree > On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java > > Lines 21 (patched) > > <https://reviews.apache.org/r/65422/diff/1/?file=1950481#file1950481line21> > > > > Use java's nonnull annotation. I've not found any "standard" annotation...I may just as well remove these markers... https://stackoverflow.com/questions/4963300/which-notnull-java-annotation-should-i-use/42695253#42695253 > On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java > > Lines 54 (patched) > > <https://reviews.apache.org/r/65422/diff/1/?file=1950484#file1950484line54> > > > > Why is this needed? this is not needed...but enables the user to set a different set of configuration during re-executions > On Feb. 16, 2018, 4:50 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java > > Lines 36 (patched) > > <https://reviews.apache.org/r/65422/diff/1/?file=1950498#file1950498line36> > > > > A flat map of operators looses hierarichal info in which operators are > > organized which is tree. So, this match needs to happen via sub-graph > > matching pattern. See SharedWorkOptimizer::areMergeable() . I will try to retain this concept for now at least; the idea is that imagine that we have N operator stats gathered; and the current plan consist of M operators; if we have only a cmp(A,B) oracle; that means we will have to do N*M comparisions; which could become really bad if N starts to become large... I'm thinking of serving the existing operator infos in a map alike fashion - at least it should be visible as one for the outside. If an operator could self-describe its whole context; then it could be match... for example a matching the textual representation of a RelNode contains all the upstream operations as well; and enables matching. It looked promising to do it; I wanted to do it with HIVE-18703 - but unfortunately there were some complications... - Zoltan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/65422/#review197649 ----------------------------------------------------------- On Jan. 30, 2018, 6:13 p.m., Zoltan Haindrich wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/65422/ > ----------------------------------------------------------- > > (Updated Jan. 30, 2018, 6:13 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Repository: hive-git > > > Description > ------- > > preview > > > Diffs > ----- > > cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java a78e0c63d7 > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b7d3e99e1a > hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatCli.java > ad31287879 > hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/HCatDriver.java > 533f0bcd6f > itests/src/test/resources/testconfiguration.properties d86ff58840 > ql/src/java/org/apache/hadoop/hive/ql/AbstractReExecDriver.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/Context.java 820fbf0f58 > ql/src/java/org/apache/hadoop/hive/ql/Driver.java 74595b00f9 > ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java 49d2bf5f33 > ql/src/java/org/apache/hadoop/hive/ql/IDriver.java 6280be0b08 > ql/src/java/org/apache/hadoop/hive/ql/ReExecOverlayDriver.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/ReOptimizeDriver.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 76e85636d1 > ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 199b181290 > ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java > 395a5f450f > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkCommonOperator.java > 8dd7cfe58c > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkEmptyKeyOperator.java > 134fc0ff0b > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkObjectHashOperator.java > 1eb72ce4d9 > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/reducesink/VectorReduceSinkUniformHashOperator.java > 384bd74686 > ql/src/java/org/apache/hadoop/hive/ql/hooks/PrivateHookContext.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > 190771ea6b > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java > cbadfa4f07 > ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 0057f0c2c6 > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/EmptyStatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/GroupTransformer.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapper.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/PlanMapperProcess.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/RuntimeStatsSource.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/SimpleRuntimeStatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/StatsSource.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/mapper/refs/OperatorRef.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java > dcf8d31eaf > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStats.java PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/stats/OperatorStatsReaderHook.java > PRE-CREATION > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFAssertTrueOOM.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java > d56002d192 > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestCounterMapping.java > PRE-CREATION > ql/src/test/org/apache/hadoop/hive/ql/plan/mapping/TestReOptimization.java > PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure.q PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure_oom.q PRE-CREATION > ql/src/test/queries/clientpositive/retry_failure_stat_changes.q > PRE-CREATION > ql/src/test/results/clientpositive/llap/retry_failure.q.out PRE-CREATION > ql/src/test/results/clientpositive/llap/retry_failure_oom.q.out > PRE-CREATION > ql/src/test/results/clientpositive/llap/retry_failure_stat_changes.q.out > PRE-CREATION > > > Diff: https://reviews.apache.org/r/65422/diff/1/ > > > Testing > ------- > > > Thanks, > > Zoltan Haindrich > >
