Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21238 )
Change subject: IMPALA-12964: Implement basic aggregation in the Calcite planner ...................................................................... IMPALA-12964: Implement basic aggregation in the Calcite planner Basic aggregation functionality is now added to the Calcite planner. The implementation of aggregation was a little tricky on the conversion from the Aggregate RelNode to the Impala Agg PlanNode. The compilation in Impala requires some AggregateInfo structures which may set up multiple internal PlanNodes. Some parts of the Analyzer are used by AggregateInfo. This usage of Analyzer puts two design goals in conflict with each other, which are: 1) Remove dependency on the Analyzer since Calcite does all the parsing and validation 2) Avoid refactoring in the first major iteration of the Calcite planner. To resolve this, a SimplifiedAnalyzer class has been created which is injected into the AggregateInfo. Some methods of the Analyzer class are overridden to avoid the non-Calcite planner analysis. The SimplifiedAnalyzer overrides two aspects of the Analyzer: 1) "Having" filter conjuncts are going to be "unassigned conjuncts". After Calcite validates and optimizes the plan, the only filter conjuncts above the aggregation will be the "having" clause, so all these conjuncts will be used in the aggregate (sidenote: optimization rules have not been pushed yet to move filters underneath the aggregate, but that will come in a push in the near future). Once the aggregate has been changed to a PlanNode, we can clear out the unassigned conjuncts. 2) Because the Aggregte PlanNodes can have multiple layers, it may be responsible for creating some TupleDescriptors and SlotDescriptors for these PlanNodes. The SlotDescriptors need to be "materialized". In the non-Calcite planner, this is done through its planning process. In the Calcite planner, the materialization can happen immediately when the PlanNode is created. So the "addSlotDescriptor" is overridden to call the parent, but then to immediately materialize the SlotDescriptor. The rest of the ImpalaAggRel is hopefully self-explanatory. The groups, aggregates, and grouping sets are extracted from the RelNodes and used in the PlanNodes. The logic to set up multiple PlanNodes and the creation of MultiAggregateInfo and AggregateInfo objects are similar to what is used in the non-Calcite planner. Change-Id: Iacf0de8ba11f0d31d73d624f0c9a91db9997cfd5 Reviewed-on: http://gerrit.cloudera.org:8080/21238 Reviewed-by: Michael Smith <[email protected]> Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ConvertToImpalaRelRules.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaAggRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalcitePhysPlanCreator.java A java/calcite-planner/src/main/java/org/apache/impala/calcite/util/SimplifiedAnalyzer.java M testdata/workloads/functional-query/queries/QueryTest/calcite.test 7 files changed, 582 insertions(+), 3 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/21238 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iacf0de8ba11f0d31d73d624f0c9a91db9997cfd5 Gerrit-Change-Number: 21238 Gerrit-PatchSet: 17 Gerrit-Owner: Steve Carlin <[email protected]> Gerrit-Reviewer: Aman Sinha <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Steve Carlin <[email protected]>
