Hello Aman Sinha, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public
Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21238
to look at the new patch set (#15).
Change subject: IMPALA-12964: Implement basic aggregation in the Calcite planner
......................................................................
IMPALA-12964: Implement basic aggregation in the Calcite planner
Basic aggregation functionality is now added to the Calcite planner.
The implementation of aggregation was a little tricky on the
conversion from the Aggregate RelNode to the Impala Agg PlanNode. The
compilation in Impala requires some AggregateInfo structures which
may set up multiple internal PlanNodes. Some parts of the Analyzer
are used by AggregateInfo.
This usage of Analyzer puts two design goals in conflict with each
other, which are:
1) Remove dependency on the Analyzer since Calcite does all the parsing
and validation
2) Avoid refactoring in the first major iteration of the Calcite planner.
To resolve this, a SimplifiedAnalyzer class has been created which is
injected into the AggregateInfo. Some methods of the Analyzer class are
overridden to avoid the non-Calcite planner analysis.
The SimplifiedAnalyzer overrides two aspects of the Analyzer:
1) "Having" filter conjuncts are going to be "unassigned conjuncts".
After Calcite validates and optimizes the plan, the only filter
conjuncts above the aggregation will be the "having" clause, so all
these conjuncts will be used in the aggregate (sidenote: optimization
rules have not been pushed yet to move filters underneath the aggregate,
but that will come in a push in the near future). Once the aggregate
has been changed to a PlanNode, we can clear out the unassigned conjuncts.
2) Because the Aggregte PlanNodes can have multiple layers, it may
be responsible for creating some TupleDescriptors and SlotDescriptors
for these PlanNodes. The SlotDescriptors need to be "materialized".
In the non-Calcite planner, this is done through its planning process.
In the Calcite planner, the materialization can happen immediately when
the PlanNode is created. So the "addSlotDescriptor" is overridden to
call the parent, but then to immediately materialize the SlotDescriptor.
The rest of the ImpalaAggRel is hopefully self-explanatory. The groups,
aggregates, and grouping sets are extracted from the RelNodes and used
in the PlanNodes. The logic to set up multiple PlanNodes and the creation
of MultiAggregateInfo and AggregateInfo objects are similar to what is
used in the non-Calcite planner.
Change-Id: Iacf0de8ba11f0d31d73d624f0c9a91db9997cfd5
---
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ConvertToImpalaRelRules.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaAggRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/util/SimplifiedAnalyzer.java
M testdata/workloads/functional-query/queries/QueryTest/calcite.test
7 files changed, 582 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/21238/15
--
To view, visit http://gerrit.cloudera.org:8080/21238
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Iacf0de8ba11f0d31d73d624f0c9a91db9997cfd5
Gerrit-Change-Number: 21238
Gerrit-PatchSet: 15
Gerrit-Owner: Steve Carlin <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
