Hello Aman Sinha, Fang-Yu Rao, Riza Suminto, Pranav Lodha, Joe McDonnell,
Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23679
to look at the new patch set (#3).
Change subject: IMPALA-14525: Calcite planner: Add support for RexSimplify
......................................................................
IMPALA-14525: Calcite planner: Add support for RexSimplify
RexSimplify is a class in Calcite that simplifies expressions
into something more optimal. It was disabled up until this point
because it converts IN clauses into a Calcite internal SEARCH
object which isn't directly supported by Impala.
This commit brings back the RexSimplify class. The SEARCH
operator is now converted into an IN operator when RexNode
objects are changed into Expr objects.
Some notes about the changes that had to be made:
- some small refactoring needed to be done in the Impala Expr
objects.
- RexSimplify is very stringent about operators that are nullable,
as there is an assert when certain operators are checked. There is
logic in the CoerceOperandShuttle that ensures the nullability is
now set correctly.
- Some duplicated logic at line 148 in CoerceOperandShuttle was removed,
(existing logic in getReturnType)
- The AnalyzedInPredicate subclass was created to avoid analysis done in
InPredicate.
- Removed ImpalaRexBuilder logic which avoided creation of the SEARCH op.
- Created ImpalaRexSimplify which extends RexSimplify. RexSimplify causes
regressions with NaN on comparisons with Double. For instance,
"where not(my_col > 30)" changes to "where my_col <= 30": The first
expression returns true when my_col is NaN and the second expression
returns false. So ImpalaRexSimplify looks for the existence of any
binary comparison operator with Double in it and avoids the simplification.
- Added ImpalaRexUtil which copies the RexUtil.expandSearch() method that
converts the SEARCH operator into non-search operators. The version here
handles the conversion to the custom Impala IN operator.
- Created an ImpalaCoreRules class. Even though RexSimplify is supported,
it is important it is run through ImpalaRexSimplify. The RexSimplify
is disabled for the SqlToRelNode converter and for all rules given by
Calcite. ImpalaCoreRules also has the benefit of having one place where
one can find all the rules used by Impala.
- Created simplify rules for the filter condition, the join condition, and
the projects in the project object.
- Changed the FilterSelectivityEstimator to get the selectivity for the
SEARCH operator.
- Added a couple of rules in the optimizer for a bug that was being exposed
when enabling the SEARCH operator. The PROJECT_JOIN_TRANSPOSE was removed
because it did not serve any purpose, as we transpose JOIN_PROJECT in the
join phase. Some other rules were added to help with pushdown predicates
like JOIN_DERIVE_IS_NOT_NULL_FILTER and JOIN_PUSH_EXPRESSIONS. And the
Simplifier rules have also been added.
- Some of the new rules caused many changes in the estimations of cardinality
and memory. The one noticeable change was using IsNullPredicate for the
IS_NULL and IS_NOT_NULL operators. Previously, these functions were using
FunctionCallExpr, and the cardinality estimation was way off.
Change-Id: I44792688f361bf15affa565e5de5709f64dcf18c
---
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/analysis/InPredicate.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceOperandShuttle.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedInPredicate.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexCallConverter.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexLiteralConverter.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaCastFunction.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaInOperator.java
D
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaRexBuilder.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaRexSimplify.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaRexUtil.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaJoinRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/CreateExprVisitor.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/util/ExprConjunctsConverter.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaCoreRules.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaFilterSimplifyRule.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaJoinSimplifyRule.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/ImpalaProjectSimplifyRule.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/FilterSelectivityEstimator.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q01.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q02.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q03.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q04.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q05.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q06.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q07.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q08.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q10a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q11.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q12.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q13.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q14a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q14b.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q15.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q16.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q17.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q18.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q19.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q20.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q21.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q22.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q23a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q23b.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q24a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q24b.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q25.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q26.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q27.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q29.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q30.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q31.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q32.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q33.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q34.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q35a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q36.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q37.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q38.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q39a.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q39b.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q40.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q41.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q42.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q43-verbose.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q43.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q44.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q45.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q46.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q47.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q48.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q49.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q50.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q51.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q52.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q53.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q54.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q55.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q56.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q57.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q58.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q59.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q60.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q61.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q62.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q63.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q64.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q65.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q66.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q67.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q68.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q69.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q70.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q71.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q72.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q73.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q74.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q75.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q76.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q77.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q78.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q79.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q80.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q81.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q82.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q83.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q84.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q85.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q86.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q87.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q88.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q89.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q90.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q91.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q92.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q93.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q94.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q95.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q96.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q97.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q98.test
M
testdata/workloads/functional-planner/queries/PlannerTest/calcite_tpcds/tpcds-q99.test
124 files changed, 48,399 insertions(+), 43,979 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/23679/3
--
To view, visit http://gerrit.cloudera.org:8080/23679
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44792688f361bf15affa565e5de5709f64dcf18c
Gerrit-Change-Number: 23679
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Fang-Yu Rao <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Pranav Lodha <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
