This is an automated email from the ASF dual-hosted git repository. joemcdonnell pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git
commit 3f0a5cc93bb85b86b8fd3d9157de13c4f0174754 Author: Steve Carlin <[email protected]> AuthorDate: Thu Apr 18 13:02:23 2024 -0700 IMPALA-13022: Added infrastructure for implicit casting of functions Ideally, at validation time, we would like Calcite to coerce function parameters and generate the proper return types that match Impala functions. Currently, Calcite comes close to hitting these needs, and any function that passes Calcite validation will be a valid function. However, while some of the types are close, we want the output of the Calcite optimizer to contain methods where all parameter types and return types exactly match what is expected by Impala at runtime. The bulk of this commit is to address this issue. There are a couple of other parts to this commit added that rely on proper types as well. This will be described as well further down in the commit message. The 'fixing' of the parameters and return types is done after optimization and before the Calcite RelNodes are translated into Impala RelNodes. The code responsible for this is under the coercenodes directory. The entry point for fixing these nodes is the method CoerceNodes.coerceNodes() which takes a RelNode input and produces a RelNode output. This method will journey through the RelNodes bottom up because a RelNode must be created with its inputs, so it makes sense to fix the inputs first. One problem within Calcite is how it generates RexLiterals. For the literal number 2, Calcite will generate an INTEGER type. However, the Impala output for this is a TINYINT. Another Calcite issue is for a string literal such as 'hello'. Calcite will generate a CHAR(5) type whereas Impala generates a string. These inconsistencies cause Impala to crash in certain situations. For instance, the query "select 'hello' union select 'goodbye'" generates a union with 2 input nodes. If we were to use the Calcite definitions, one would have type of char(5) and the other would have type of char(7), which would cause a crash. Also, eventually when CTAS statements are implemented, we need the types to match Impala's native type. Most of the Calcite RelNode types need some sort of correction due to these issues. Join, Filter and Project nodes may have expressions that need fixing. The CoerceOperandShuttle class helps navigating through the RexNode function structure for these. The Aggregate class may require an underlying Project class where the explanation is detailed in the processAggNode method. The Union node also will generate underlying Project nodes for casting. The Values node creates a Project above itself since Calcite will not allow generation of a RexLiteral of string type directly, so it needs to be cast. In addition to these changes, other changes that relied on casting issues have been added. The or/and/case operator is now supported. 'Or' and 'and' worked before, but not if there were more than 2 or/and conditions grouped together. Case requires all return types to match and required some special logic. These functions required a little extra support since they have variable arguments. Change-Id: I13a349673f185463276ad7ddb83f0cfc2d73218c Reviewed-on: http://gerrit.cloudera.org:8080/21335 Reviewed-by: Michael Smith <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Joe McDonnell <[email protected]> --- .../impala/calcite/coercenodes/CoerceNodes.java | 665 +++++++++++++++++++++ .../calcite/coercenodes/CoerceOperandShuttle.java | 280 +++++++++ .../impala/calcite/functions/AnalyzedCaseExpr.java | 69 +++ .../impala/calcite/functions/AnalyzedCastExpr.java | 55 ++ .../functions/AnalyzedFunctionCallExpr.java | 2 +- .../impala/calcite/functions/FunctionResolver.java | 124 +++- .../calcite/functions/ImplicitTypeChecker.java | 158 +++++ .../impala/calcite/functions/RexCallConverter.java | 80 ++- .../calcite/functions/RexLiteralConverter.java | 6 +- .../calcite/operators/ImpalaConvertletTable.java | 50 ++ .../operators/ImpalaCustomOperatorTable.java | 201 +++++++ .../impala/calcite/operators/ImpalaOperator.java | 10 +- .../calcite/operators/ImpalaOperatorTable.java | 15 +- .../impala/calcite/rel/node/ImpalaAggRel.java | 2 +- .../impala/calcite/rel/node/ImpalaJoinRel.java | 2 +- .../impala/calcite/rel/node/ImpalaPlanRel.java | 37 ++ .../impala/calcite/rel/node/ImpalaValuesRel.java | 37 +- .../calcite/rel/node/ParentPlanRelContext.java | 14 + .../impala/calcite/service/CalciteOptimizer.java | 8 +- .../calcite/service/CalciteRelNodeConverter.java | 4 +- .../impala/calcite/type/ImpalaTypeConverter.java | 63 ++ .../impala/calcite/type/ImpalaTypeSystemImpl.java | 114 +++- .../java/org/apache/impala/CompatibilityTest.java | 96 +++ .../queries/QueryTest/calcite.test | 303 +++++++++- 24 files changed, 2324 insertions(+), 71 deletions(-) diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceNodes.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceNodes.java new file mode 100644 index 000000000..01dcb1f1d --- /dev/null +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceNodes.java @@ -0,0 +1,665 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.impala.calcite.coercenodes; + +import com.google.common.base.Preconditions; +import com.google.common.collect.ImmutableList; +import org.apache.calcite.rel.RelNode; +import org.apache.calcite.rel.core.AggregateCall; +import org.apache.calcite.rel.core.RelFactories; +import org.apache.calcite.rel.logical.LogicalAggregate; +import org.apache.calcite.rel.logical.LogicalFilter; +import org.apache.calcite.rel.logical.LogicalJoin; +import org.apache.calcite.rel.logical.LogicalProject; +import org.apache.calcite.rel.logical.LogicalUnion; +import org.apache.calcite.rel.logical.LogicalValues; +import org.apache.calcite.rel.type.RelDataType; +import org.apache.calcite.rel.type.RelDataTypeField; +import org.apache.calcite.rel.type.RelDataTypeFactory; +import org.apache.calcite.rex.RexBuilder; +import org.apache.calcite.rex.RexInputRef; +import org.apache.calcite.rex.RexLiteral; +import org.apache.calcite.rex.RexNode; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.type.SqlTypeName; +import org.apache.calcite.util.Util; +import org.apache.impala.calcite.functions.FunctionResolver; +import org.apache.impala.calcite.rel.node.ImpalaPlanRel; +import org.apache.impala.calcite.type.ImpalaTypeConverter; +import org.apache.impala.catalog.Function; +import org.apache.impala.catalog.Type; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; +import java.util.stream.Collectors; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * CoerceNodes is responsible for coercing Calcite RelNode objects into + * RelNodes that conform to Impala requirements. The entry point is via + * the coerceNodes method. + * + * The long term goal would be to have Calcite validation be responsible + * for creating the correct parameter types and return types for functions + * at validation time. They do have type coercion capabilities at validation + * time, but Calcite has some shortcomings. These shortcomings are: + * - Integer literals are always created as type INTEGER. In Impala, + * the number 2 would be treated as a TINYINT + * - Char literals are treated as CHAR<X> in Calcite. In Impala, they + * are treated as string types + * - We try to use Calcite operators whenever possible since they are + * used and created throughout the analysis and optimization phases. This + * can possibly differ from Impala function checks so it is nice to have + * the override capabilities. + * + * Until we can fix these issues at validation time, we implement the coercion + * of RelNodes and RexNodes at the end of optimization to ensure that Impala + * has the correction functions. + */ +public class CoerceNodes{ + protected static final Logger LOG = + LoggerFactory.getLogger(CoerceNodes.class.getName()); + + /** + * coerceNodes is the entry point method (the only public method in this class). + */ + public static RelNode coerceNodes(RelNode relNode, RexBuilder rexBuilder) { + RelNode newRelNode = coerceNodesInternal(relNode, rexBuilder); + return newRelNode != null ? newRelNode : relNode; + } + + /** + * coerceNodesInternal is a recursive walkthrough of the RelNode tree. The bottom + * levels are processed first. + */ + private static RelNode coerceNodesInternal(RelNode relNode, RexBuilder rexBuilder) { + boolean isInputChanged = false; + List<RelNode> newInputs = new ArrayList<>(); + for (RelNode input : relNode.getInputs()) { + RelNode changedInput = coerceNodesInternal(input, rexBuilder); + isInputChanged |= (changedInput != input); + newInputs.add(changedInput); + } + + switch (ImpalaPlanRel.getRelNodeType(relNode)) { + case AGGREGATE: + return processAggNode(relNode, newInputs, rexBuilder, isInputChanged); + case FILTER: + return processFilterNode(relNode, newInputs, rexBuilder, isInputChanged); + case JOIN: + return processJoinNode(relNode, newInputs, rexBuilder, isInputChanged); + case PROJECT: + return processProjectNode(relNode, newInputs, rexBuilder, isInputChanged); + case SORT: + return processSortNode(relNode, newInputs, rexBuilder, isInputChanged); + case UNION: + return processUnionNode(relNode, newInputs, rexBuilder, isInputChanged); + case VALUES: + return processValuesNode(relNode, newInputs, rexBuilder, isInputChanged); + case HDFSSCAN: + // HDFS Scan node will never need coercing. + return relNode; + } + + throw new RuntimeException("Unrecognized RelNode: " + relNode); + } + + /** + * processFilterNode: Checks and coerces any method in the condition. + */ + private static RelNode processFilterNode(RelNode relNode, List<RelNode> inputs, + RexBuilder rexBuilder, boolean isInputChanged) { + final LogicalFilter filter = (LogicalFilter) relNode; + + RexNode condition = filter.getCondition(); + + List<RexNode> changedRexNodes = + processRexNodes(relNode, inputs, ImmutableList.of(condition)); + + // If any RexNode was changed or any RelNode input was changed, we need + // to create a new RelNode. Otherwise, just return existing RelNode. + if (changedRexNodes == null && !isInputChanged) { + return relNode; + } + + RexNode newCondition = (changedRexNodes == null) ? condition : changedRexNodes.get(0); + return filter.copy(filter.getTraitSet(), inputs.get(0), newCondition); + } + + /** + * processJoinNode: Checks and coerces any method in the condition. + */ + private static RelNode processJoinNode(RelNode relNode, List<RelNode> inputs, + RexBuilder rexBuilder, boolean isInputChanged) { + final LogicalJoin join = (LogicalJoin) relNode; + + RexNode condition = join.getCondition(); + + List<RexNode> changedRexNodes = + processRexNodes(relNode, inputs, ImmutableList.of(condition)); + + // If any RexNode was changed or any RelNode input was changed, we need + // to create a new RelNode. Otherwise, just return existing RelNode. + if (changedRexNodes == null && !isInputChanged) { + return relNode; + } + + RexNode newCondition = (changedRexNodes == null) ? condition : changedRexNodes.get(0); + + return join.copy(join.getTraitSet(), newCondition, inputs.get(0), inputs.get(1), + join.getJoinType(), join.isSemiJoinDone()); + } + + /** + * processProjectNode: Checks and coerces any method in the projects. + */ + private static RelNode processProjectNode(RelNode relNode, List<RelNode> inputs, + RexBuilder rexBuilder, boolean isInputChanged) { + final LogicalProject project = (LogicalProject) relNode; + + List<RexNode> projects = project.getProjects(); + List<RexNode> changedRexNodes = processRexNodes(relNode, inputs, projects); + + // If any RexNode was changed or any RelNode input was changed, we need + // to create a new RelNode. Otherwise, just return existing RelNode. + if (changedRexNodes == null && !isInputChanged) { + return relNode; + } + + List<RexNode> newProjects = changedRexNodes == null ? projects : changedRexNodes; + + RelDataTypeFactory factory = rexBuilder.getTypeFactory(); + List<RelDataType> typeList = Util.transform(newProjects, RexNode::getType); + RelDataType rowType = + factory.createStructType(typeList, project.getRowType().getFieldNames()); + return project.copy(project.getTraitSet(), inputs.get(0), newProjects, rowType); + } + + /** + * processSortNode: recreates sort node if an input was changed. + */ + private static RelNode processSortNode(RelNode relNode, List<RelNode> inputs, + RexBuilder rexBuilder, boolean isInputChanged) { + return isInputChanged ? relNode.copy(relNode.getTraitSet(), inputs) : relNode; + } + + /** + * processAggNode: Checks if there is an existing function signature for + * the aggregation call. If a coercion is needed, we need to do some processing. + * The Aggregate RelNode is not allowed to contain a casted expression. + * Instead, a Project RelNode is placed as an input to the Aggregate RelNode. The + * Project RelNode will have its first <n> fields map directly into the original input + * node of the Aggregate. Additional 'cast' RexCalls will be added into the Project + * and the AggregateCall will point into these input columns. + */ + private static RelNode processAggNode(RelNode relNode, List<RelNode> inputs, + RexBuilder rexBuilder, boolean isInputChanged) { + final LogicalAggregate agg = (LogicalAggregate) relNode; + + int numInputFields = agg.getInput(0).getRowType().getFieldCount(); + + // List of aggCalls that are either copied over from the original agg node + // or transformed because of coercion. + List<AggregateCall> transformedAggCallList = new ArrayList<>(); + + // New fields and fieldnames that are needed because of coercion. + List<RexNode> newProjectFields = new ArrayList<>(); + List<String> newProjectFieldNames = new ArrayList<>(); + + for (AggregateCall aggCall : agg.getAggCallList()) { + List<RelDataType> operandTypes = getOperandTypes(inputs.get(0), aggCall); + // check to see if the current agg call matches an existing Impala + // function without casting any operands + if (matchesSignature(aggCall, operandTypes)) { + // just use original aggCall + transformedAggCallList.add(aggCall); + continue; + } + + // if here, need to transform + List<RelDataType> newOperandTypes = getCastedOperandTypes(aggCall, operandTypes); + + // last parameter is the starting point for new project fields added + // (see method comment for details). + AggregateCall newAggCall = getNewAggCall(aggCall, operandTypes, newOperandTypes, + numInputFields + newProjectFields.size()); + transformedAggCallList.add(newAggCall); + + // Add the newly cast fields into a soon to be generated underlying Project for + // this AggCall + newProjectFields.addAll(getNewProjectFields( + agg.getCluster().getRexBuilder(), aggCall, operandTypes, newOperandTypes)); + newProjectFieldNames.addAll(getNewProjectFieldNames( + agg.getInput(0), aggCall, operandTypes, newOperandTypes)); + } + + if (newProjectFields.isEmpty()) { + return isInputChanged ? relNode.copy(relNode.getTraitSet(), inputs) : relNode; + } + + RelNode project = + createProject(inputs.get(0), newProjectFields, newProjectFieldNames); + return LogicalAggregate.create(project, agg.getGroupSet(), agg.getGroupSets(), + transformedAggCallList); + } + + /** + * processUnionNode: Checks the types of all columns in the inputs. They need + * to match. If they do not match across all inputs, a Project node is created + * to cast the column to a common type. The Union node is then recreated with + * the new common type. + */ + private static RelNode processUnionNode(RelNode relNode, List<RelNode> inputs, + RexBuilder rexBuilder, boolean isInputChanged) { + final LogicalUnion union = (LogicalUnion) relNode; + + // no work to be done if no input changed and there's only one input. + if (!isInputChanged && inputs.size() == 1) { + return relNode; + } + + // Calculate the common row types for all the columns. + List<RelDataType> commonRowType = getCompatibleRowType(relNode, inputs, rexBuilder); + + // Check to see if the union rowtype is different from the common row type calculated. + // The boolean inputsChanged is used to determine if we need to recreate the Union + // node, so if any column type is different, we set the value to true. + boolean inputsChanged = isInputChanged || + haveTypesChanged(commonRowType, union.getRowType().getFieldList()); + + List<RelNode> changedRelNodes = new ArrayList<>(); + for (RelNode input : inputs) { + // getChangedInput returns the same RelNode if the RelNode has not changed. + RelNode changedRelNode = getChangedUnionInput(input, commonRowType, rexBuilder); + boolean inputChanged = !changedRelNode.equals(input); + changedRelNodes.add(inputChanged ? changedRelNode : input); + inputsChanged |= inputChanged; + } + + return inputsChanged ? LogicalUnion.create(changedRelNodes, union.all) : relNode; + } + + /** + * processValuesNode: Coerces Value node literals (numerics and strings). For + * strings, an intermediate Project node needs to be created (see comment below). + */ + private static RelNode processValuesNode(RelNode relNode, List<RelNode> inputs, + RexBuilder rexBuilder, boolean isInputChanged) { + final LogicalValues values = (LogicalValues) relNode; + if (values.getTuples().size() == 0) { + return relNode; + } + + int nColumns = values.getRowType().getFieldList().size(); + // initialize list to have null values for all columns + List<RelDataType> relDataTypes = Arrays.asList(new RelDataType[nColumns]); + + boolean needProject = false; + for (List<RexLiteral> tuple : values.getTuples()) { + List<RexNode> rexNodes = castToRexNodeList(tuple); + List<RexNode> changedRexNodes = processRexNodes(relNode, inputs, rexNodes); + if (changedRexNodes == null) { + continue; + } + needProject = true; + Preconditions.checkState(changedRexNodes.size() == relDataTypes.size()); + for (int i = 0; i < changedRexNodes.size(); ++i) { + if (changedRexNodes.get(i) != null) { + Preconditions.checkState(changedRexNodes.get(i).getKind() == SqlKind.CAST || + changedRexNodes.get(i) instanceof RexLiteral); + } + // if changedRexNodes.get(i) is something other than null, the type needs to + // be coerced. We want to take the tightest type we can. The current tightest + // type is in the relDataTypes.get(i). On initialization, it is set to null, + // so if this is the first row that has a coerced type for the ith column, + // the tightest type will be the current changedRexNodes.get(i).getType() type. + relDataTypes.set(i, getCompatibleDataType( + relDataTypes.get(i), changedRexNodes.get(i).getType(), rexBuilder)); + } + } + + if (!needProject) { + return relNode; + } + + // Need to create a project node on top of the values: A project node + // does not add any overhead performance-wise since it doesn't create + // a new node. However, it is needed here because Calcite creates string + // literals as CHAR type and Impala requires a STRING type. The project + // node creates this casting which will get removed when converting to the + // Impala Expr object (where the RelNodes get converted to the physical layer). + List<RexNode> projects = new ArrayList<>(); + + for (int i = 0; i < relDataTypes.size(); ++i) { + RexInputRef inputRef = rexBuilder.makeInputRef(values, i); + + RexNode project = (relDataTypes.get(i) != null) + ? rexBuilder.makeCast(relDataTypes.get(i), inputRef) + : inputRef; + projects.add(project); + } + + return LogicalProject.create(values, new ArrayList<>(), projects, + values.getRowType().getFieldNames()); + } + + ////////////////////////////////////////////////////////// + // General method helpers + ////////////////////////////////////////////////////////// + + /** + * processRexNodes is a shared method that coerces the list of RexNodes passed in. + */ + private static List<RexNode> processRexNodes(RelNode relNode, List<RelNode> inputs, + List<RexNode> rexNodes) { + CoerceOperandShuttle shuttle = new CoerceOperandShuttle( + relNode.getCluster().getTypeFactory(), + relNode.getCluster().getRexBuilder(), inputs); + List<RexNode> changedRexNodes = new ArrayList<>(); + boolean rexNodeChanged = false; + for (RexNode rexNode : rexNodes) { + RexNode changedRexNode = shuttle.apply(rexNode); + changedRexNodes.add(changedRexNode); + rexNodeChanged |= (changedRexNode != rexNode); + } + return rexNodeChanged ? changedRexNodes : null; + } + + /** + * method to translate a RexLiteral list to a RexNode list + */ + private static List<RexNode> castToRexNodeList(List<RexLiteral> literalList) { + Class<RexNode> clazz = RexNode.class; + return literalList.stream() + .map(clazz::cast) + .collect(Collectors.toList()); + } + + /** + * compare 2 datatypes and return a datatype that is compatible to both of them. + */ + private static RelDataType getCompatibleDataType(RelDataType dt1, + RelDataType dt2, RexBuilder rexBuilder) { + if (dt1 == null) { + return dt2; + } + + if (dt2 == null) { + return dt1; + } + + return ImpalaTypeConverter.getCompatibleType( + ImmutableList.of(dt1, dt2), rexBuilder.getTypeFactory()); + } + + /** + * getCompatibleDataType takes a list of inputs with multiple column types + * and returns a list of types that are common (and castable) for all + * the inputs. + */ + private static List<RelDataType> getCompatibleRowType(RelNode origRelNode, + List<RelNode> inputs, RexBuilder rexBuilder) { + RelDataTypeFactory factory = rexBuilder.getTypeFactory(); + + List<RelDataType> finalTypes = new ArrayList<>(); + for (RelDataTypeField field : inputs.get(0).getRowType().getFieldList()) { + finalTypes.add(field.getType()); + } + + for (int j = 1; j < inputs.size(); ++j ) { + RelNode input = inputs.get(j); + for (int i = 0; i < input.getRowType().getFieldList().size(); ++i) { + RelDataType type0 = finalTypes.get(i); + RelDataType type1 = input.getRowType().getFieldList().get(i).getType(); + finalTypes.set(i, + ImpalaTypeConverter.getCompatibleType(type0, type1, factory)); + } + } + return finalTypes; + } + + /** + * getChangedUnionInput takes a RelNode and a new "common" row type. If any column + * type in the common row type is different from a column in the RelNode, + * a Project node will be created that casts the offending columns into the + * common type. + */ + private static RelNode getChangedUnionInput(RelNode relNode, + List<RelDataType> commonRowType, RexBuilder rexBuilder) { + boolean changed = false; + List<RexNode> projects = new ArrayList<>(); + for (int i = 0; i < relNode.getRowType().getFieldList().size(); ++i) { + RexInputRef inputRef = rexBuilder.makeInputRef(relNode, i); + RelDataType inputType = relNode.getRowType().getFieldList().get(i).getType(); + boolean projectChanged = !inputType.equals(commonRowType.get(i)); + RexNode project = projectChanged + ? rexBuilder.makeCast(commonRowType.get(i), inputRef) + : inputRef; + projects.add(project); + changed |= projectChanged; + } + + if (!changed) { + return relNode; + } + + RelDataTypeFactory factory = rexBuilder.getTypeFactory(); + RelDataType rowType = + factory.createStructType(commonRowType, relNode.getRowType().getFieldNames()); + return LogicalProject.create(relNode, new ArrayList<>(), projects, rowType); + } + + /** + * haveTypesChanged is a helper method to determine if any type in a + * RelDataTypeField list is different from a type in a common row type. + */ + private static boolean haveTypesChanged(List<RelDataType> commonRowType, + List<RelDataTypeField> fields) { + Preconditions.checkState(commonRowType.size() == fields.size()); + for (int i = 0; i < commonRowType.size(); ++i) { + if (!commonRowType.get(i).equals(fields.get(i).getType())) { + return true; + } + } + return false; + } + + private static boolean matchesSignature(AggregateCall aggCall, + List<RelDataType> operandTypes) { + // The single_value function is what Calcite uses for cardinality checks. Impala + // creates the CardinalityCheckNode when this function is translated. So the + // single_value function is not a normal agg function that exists as an Impala + // function. + // We just return true since a matched signature means that no more processing or + // coercing needs to be done. + if (aggCall.getAggregation().getName().toLowerCase().equals("single_value")) { + return true; + } + + // quick pass to check for nulls. Since Impala does not contain a null as + // an input type, return false because the signature will not match. + for (RelDataType relDataType : operandTypes) { + if (relDataType.getSqlTypeName() == SqlTypeName.NULL) { + return false; + } + } + + // Look for a function match. If found, no need to coerce + Function fn = FunctionResolver.getExactFunction(aggCall.getAggregation().getName(), + operandTypes); + + if (fn == null) { + return false; + } + + // One last check to make sure the return types match. + RelDataType retType = ImpalaTypeConverter.getRelDataType(fn.getReturnType()); + return retType.getSqlTypeName().equals(aggCall.getType().getSqlTypeName()); + } + + /** + * getOperandTypes for an AggCall. Calcite only keeps the input ref number in the + * AggCall, so to get the actual operand type, we need to go to the input RelNode. + */ + private static List<RelDataType> getOperandTypes(RelNode input, AggregateCall aggCall) { + List<RelDataType> operandTypes = new ArrayList<>(); + for (Integer i : aggCall.getArgList()) { + // add the type of then ith column in the input + operandTypes.add(input.getRowType().getFieldList().get(i).getType()); + } + return operandTypes; + } + + /** + * getCastedOperandTypes takes an AggregateCall with a list of its + * parameters types and returns a corresponding list of the types + * in a matching Impala function signature. + */ + private static List<RelDataType> getCastedOperandTypes(AggregateCall aggCall, + List<RelDataType> operandTypes) { + // Get the Impala function. Getting the "supertype" function will retrieve + // the closest function where operands may be cast. + Function fn = FunctionResolver.getSupertypeFunction( + aggCall.getAggregation().getName(), operandTypes); + Preconditions.checkNotNull(fn, "Could not find matching functions for " + + aggCall.getAggregation().getName()); + RelDataType retType = ImpalaTypeConverter.getRelDataType(fn.getReturnType()); + + // Not changing return type, they should be the same. The code will get more + // complicated if this has to change. + Preconditions.checkState( + retType.getSqlTypeName().equals(aggCall.getType().getSqlTypeName()) || + aggCall.getType().getSqlTypeName().equals(SqlTypeName.NULL)); + + List<RelDataType> newOperandTypes = new ArrayList<>(); + for (int i = 0; i < operandTypes.size(); ++i) { + Type t = (i < fn.getArgs().length) + ? fn.getArgs()[i] + : fn.getArgs()[fn.getArgs().length - 1]; + newOperandTypes.add(ImpalaTypeConverter.getRelDataType(t)); + } + return newOperandTypes; + } + + /** + * genNewAggCall generates a new Aggregate call based on new operand + * types that are needed. The AggregateCall method does not contain + * any type information nor does it contain a RexCall. It only contains + * an input ref number from its input node. So the AggregateCall needs + * to change in steps. If the AggregateCall requires a casted type, we + * will need to create an underlying Project column which creates this + * cast. We can't just change the column in case other AggregateCalls + * depend on that input ref (it would make the logic more complex to + * handle it that way). + * + * The numProjects passed in represents the already existing number of + * project columns. Any newly created input will start with a reference + * number from this value. + */ + private static AggregateCall getNewAggCall(AggregateCall aggCall, + List<RelDataType> operandTypes, List<RelDataType> newOperandTypes, + int numProjects) { + List<Integer> newArgList = new ArrayList<>(); + Preconditions.checkState(aggCall.getArgList().size() == operandTypes.size()); + Preconditions.checkState(operandTypes.size() == newOperandTypes.size()); + for (int i = 0; i < operandTypes.size(); ++i) { + // If the new needed operand type is the same as the previous operand + // type, we use the previous operand type which will be the same + // index number in the underlying Project. If a new one is needed, + // the underlying Project will be the next available index. + boolean typesEqual = areSqlTypesEqual(operandTypes.get(i), newOperandTypes.get(i)); + int newArg = typesEqual + ? aggCall.getArgList().get(i) + : numProjects++; + newArgList.add(newArg); + } + return aggCall.withArgList(newArgList); + } + + /** + * genNewProjectFields works in tandem with getNewAggCall, but this creates the + * actual RexNodes used in the underlying Project. We iterate through both the + * old types and the new types. If we see the corresponding types don't match, + * for the ith value, we create a RexNode casting the ith input ref into the new + * type. + */ + private static List<RexNode> getNewProjectFields(RexBuilder rexBuilder, + AggregateCall aggCall, List<RelDataType> operandTypes, + List<RelDataType> newOperandTypes) { + List<RexNode> newProjects = new ArrayList<>(); + for (int i = 0; i < operandTypes.size(); ++i) { + if (!areSqlTypesEqual(operandTypes.get(i), newOperandTypes.get(i))) { + RexInputRef inputRef = rexBuilder.makeInputRef( + operandTypes.get(i), aggCall.getArgList().get(i)); + RexNode newProject = rexBuilder.makeCast(newOperandTypes.get(i), inputRef); + newProjects.add(newProject); + } + } + return newProjects; + } + + /** + * genNewProjectFieldNames works the same as getNewProjectFields, but the + * names are generated for the new fields with a "cast_" prefix. + */ + private static List<String> getNewProjectFieldNames(RelNode input, + AggregateCall aggCall, List<RelDataType> operandTypes, + List<RelDataType> newOperandTypes) { + List<String> newNames = new ArrayList<>(); + for (int i = 0; i < operandTypes.size(); ++i) { + if (!areSqlTypesEqual(operandTypes.get(i), newOperandTypes.get(i))) { + String precastFieldName = + input.getRowType().getFieldNames().get(aggCall.getArgList().get(i)); + newNames.add("cast_" + precastFieldName); + } + } + return newNames; + } + + private static boolean areSqlTypesEqual(RelDataType r1, RelDataType r2) { + return r1.getSqlTypeName().equals(r2.getSqlTypeName()); + } + + /** + * createProject creates the underlying Project node under the Aggregate + * for when there are added casts. The first <n> fields match the original + * input and we just need to create a matching RexInputRef aligning the columns. + * After the first <n> input columns, we add the new casted columns to the + * Project. + */ + private static RelNode createProject(RelNode input, List<RexNode> newProjectFields, + List<String> newFieldNames) { + List<RexNode> projects = new ArrayList<>(); + RexBuilder rexBuilder = input.getCluster().getRexBuilder(); + for (int i = 0; i < input.getRowType().getFieldCount(); ++i) { + projects.add(rexBuilder.makeInputRef( + input.getRowType().getFieldList().get(i).getType(), i)); + } + List<String> fieldNames = + new ArrayList<>(input.getRowType().getFieldNames()); + projects.addAll(newProjectFields); + fieldNames.addAll(newFieldNames); + + return RelFactories.DEFAULT_PROJECT_FACTORY.createProject(input, + new ArrayList<>(), projects, fieldNames, new HashSet<>()); + } +} diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceOperandShuttle.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceOperandShuttle.java new file mode 100644 index 000000000..fb73c4bb6 --- /dev/null +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/coercenodes/CoerceOperandShuttle.java @@ -0,0 +1,280 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.impala.calcite.coercenodes; + +import com.google.common.base.Preconditions; +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableSet; +import org.apache.calcite.rel.RelNode; +import org.apache.calcite.rex.RexBuilder; +import org.apache.calcite.rex.RexCall; +import org.apache.calcite.rex.RexInputRef; +import org.apache.calcite.rex.RexLiteral; +import org.apache.calcite.rex.RexNode; +import org.apache.calcite.rex.RexShuttle; +import org.apache.calcite.rex.RexUtil; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.type.SqlTypeName; +import org.apache.calcite.rel.type.RelDataType; +import org.apache.calcite.rel.type.RelDataTypeField; +import org.apache.calcite.rel.type.RelDataTypeFactory; +import org.apache.calcite.util.Util; +import org.apache.impala.catalog.Function; +import org.apache.impala.catalog.ScalarType; +import org.apache.impala.catalog.Type; +import org.apache.impala.calcite.functions.FunctionResolver; +import org.apache.impala.calcite.functions.ImplicitTypeChecker; +import org.apache.impala.calcite.type.ImpalaTypeConverter; + +import java.math.BigDecimal; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * CoerceOperandShuttle is a RexShuttle that walks through a RexNode and changes + * it to match a function signature within Impala. It also is responsible for + * changing RexLiteral types. It changes all CHAR literal types to STRING literal + * types. It changes Integer numeric literals to the smallest type which can hold the + * integer (e.g. 2 gets changed from INTEGER to TINYINT). It also changes RexInputRefs + * to match its input type. + * + * One small added responsibility is to take the "Sarg" call and call the Calcite + * RexUtil.expandSearch method and expands it to something Impala understands. There + * are various Impala rules that create this "Sarg" method. Impala-13369 has been filed to + * investigate if there is a more optimal Impala function that can be used. + */ + +public class CoerceOperandShuttle extends RexShuttle { + protected static final Logger LOG = + LoggerFactory.getLogger(CoerceOperandShuttle.class.getName()); + private final RelDataTypeFactory factory; + private final RexBuilder rexBuilder; + private final List<RelNode> inputs; + + public static Set<SqlKind> NO_CASTING_NEEDED = + ImmutableSet.<SqlKind> builder() + // Cast doesn't need any operand casting because it is already a cast. + .add(SqlKind.CAST) + // OR and AND operands are always boolean. Just skip processing rather + // than remove the varargs (since these can have many operands and Impala's + // signature only has 2 operands) + .add(SqlKind.OR) + .add(SqlKind.AND) + .build(); + + public CoerceOperandShuttle(RelDataTypeFactory factory, RexBuilder rexBuilder, + List<RelNode> inputs) { + this.factory = factory; + this.rexBuilder = rexBuilder; + this.inputs = inputs; + } + + @Override + public RexNode visitCall(RexCall call) { + + // Eliminate the "Sarg" function which is unknown to Impala. + // TODO: this is kinda hacky. It would be better if Impala can handle this + // directly, so this needs investigation. + if (call.getOperator().getKind().equals(SqlKind.SEARCH)) { + return visitCall((RexCall) RexUtil.expandSearch(rexBuilder, null, call)); + } + + // recursively call all embedded RexCalls first + RexCall castedOperandsCall = (RexCall) super.visitCall(call); + + // Certain operators will never need casting for their operands. + if (NO_CASTING_NEEDED.contains(castedOperandsCall.getOperator().getKind())) { + return castedOperandsCall; + } + + Function fn = FunctionResolver.getSupertypeFunction(castedOperandsCall); + + if (fn == null) { + throw new RuntimeException("Could not find a matching signature for call " + + call); + } + + RelDataType retType = ImpalaTypeConverter.getRelDataType(fn.getReturnType()); + + // This code does not handle changes in the return type when the Calcite + // function is not a decimal but the function resolves to a function that + // returns a decimal type. The Decimal type from the function resolver would + // have to calculate the precision and scale based on operand types. If + // necessary, this code should be added later. + Preconditions.checkState(retType.getSqlTypeName() != SqlTypeName.DECIMAL || + castedOperandsCall.getType().getSqlTypeName() == SqlTypeName.DECIMAL); + + // So if the original return type is Decimal and the function resolves to + // decimal, the precision and scale are saved from the original function. + if (retType.getSqlTypeName().equals(SqlTypeName.DECIMAL)) { + retType = castedOperandsCall.getType(); + } + + List<RexNode> newOperands = + getCastedArgTypes(fn, castedOperandsCall.getOperands(), factory, rexBuilder); + + // keep the original call if nothing changed, else build a new RexCall. + return retType.equals(castedOperandsCall.getType()) + && newOperands.equals(castedOperandsCall.getOperands()) + ? castedOperandsCall + : (RexCall) rexBuilder.makeCall(retType, castedOperandsCall.getOperator(), + newOperands); + } + + @Override + public RexNode visitLiteral(RexLiteral literal) { + // Coerce CHAR literal types into STRING + if (literal.getType().getSqlTypeName().equals(SqlTypeName.CHAR)) { + return rexBuilder.makeLiteral(RexLiteral.stringValue(literal), + ImpalaTypeConverter.getRelDataType(Type.STRING), true, true); + } + + // Coerce INTEGER literal types into the smallest possible Numeric type + if (literal.getType().getSqlTypeName().equals(SqlTypeName.INTEGER)) { + BigDecimal bd0 = literal.getValueAs(BigDecimal.class); + RelDataType type = ImpalaTypeConverter.getLiteralDataType(bd0, literal.getType()); + return rexBuilder.makeLiteral(bd0, type); + } + return literal; + } + + @Override + public RexNode visitInputRef(RexInputRef inputRef) { + // Adjust the InputRef type if it changed + RelDataType inputRefIndexType = getInputRefIndexType(inputs, inputRef.getIndex()); + + return inputRef.getType().equals(inputRefIndexType) + ? inputRef + : rexBuilder.makeInputRef(inputRefIndexType, inputRef.getIndex()); + } + + /** + * Handle getting the type of index. If there is only one input, then we + * just use the index value to get the type. If there are two inputs, + * then the second input's index value starts at the number which is + * the size of the first input. + */ + private RelDataType getInputRefIndexType(List<RelNode> inputs, int index) { + if (inputs.size() == 1) { + return inputs.get(0).getRowType().getFieldList().get(index).getType(); + } + + // currently only works for joins which have 2 inputs + Preconditions.checkState(inputs.size() == 2); + List<RelDataTypeField> leftFieldList = inputs.get(0).getRowType().getFieldList(); + + // If the index number is greater than or equal to the number of fields in + // the left input, it must be in the right input. + if (index < leftFieldList.size()) { + return leftFieldList.get(index).getType(); + } + int rightIndex = index - leftFieldList.size(); + return inputs.get(1).getRowType().getFieldList().get(rightIndex).getType(); + } + + /** + * Return a list of the operands, casting whenever needed. + */ + private static List<RexNode> getCastedArgTypes(Function fn, List<RexNode> operands, + RelDataTypeFactory factory, RexBuilder rexBuilder) { + + List<RelDataType> argTypes = Util.transform(operands, RexNode::getType); + List<RexNode> newOperands = new ArrayList<>(); + // The "Case" operator is special because the operands alternate between + // "when" and "then" conditions, and the "when" conditions are always + // boolean, so they don't need casting. + boolean isCaseFunction = isCaseFunction(fn); + boolean castedOperand = false; + for (int i = 0; i < argTypes.size(); ++i) { + if (isCaseFunction && + FunctionResolver.shouldSkipOperandForCase(argTypes.size(), i)) { + // if skipped, we leave the operand type as/is. + newOperands.add(operands.get(i)); + continue; + } + // if there are varargs, the last arg in the signature will match all + // remaining args. + int sigIndex = getArgIndex(fn, i, isCaseFunction); + RexNode operand = castOperand(operands.get(i), fn.getArgs()[sigIndex], + factory, rexBuilder); + Preconditions.checkNotNull(operand); + newOperands.add(operand); + if (!operands.get(i).equals(operand)) { + castedOperand = true; + } + } + + return castedOperand ? newOperands : operands; + } + + /** + * Return the argIndex. If it's a case statement, the index is always 0 + * If there are varargs, the last index returns if the "i" value passed + * in overflows the size of the operands. + */ + private static int getArgIndex(Function fn, int i, boolean isCaseFn) { + if (isCaseFn) { + return 0; + } + + return Math.min(i, fn.getNumArgs() - 1); + } + + private static boolean isCaseFunction(Function fn) { + return fn.functionName().equals("case"); + } + + /** + * castOperand takes a RexNode and a toType and returns the RexNode + * with a potential cast wrapper. If the types match, then the + * original RexNode is returned. If the toType is an incompatible + * type, this method returns null. + */ + private static RexNode castOperand(RexNode node, Type toImpalaType, + RelDataTypeFactory factory, RexBuilder rexBuilder) { + RelDataType fromType = node.getType(); + + RelDataType toType = ImpalaTypeConverter.getRelDataType(toImpalaType); + + // No need to cast if types are the same + if (fromType.getSqlTypeName().equals(toType.getSqlTypeName())) { + return node; + } + + if (fromType.getSqlTypeName().equals(SqlTypeName.NULL)) { + return rexBuilder.makeCast(toType, node); + } + + if (!ImplicitTypeChecker.supportsImplicitCasting(fromType, toType)) { + return null; + } + + // Integer based type needs special conversion to Decimal types based on the + // size of the type of Integer (e.g. TINYINT, SMALLINT, etc...) + if (toType.getSqlTypeName().equals(SqlTypeName.DECIMAL)) { + ScalarType impalaType = (ScalarType) ImpalaTypeConverter.createImpalaType(fromType); + ScalarType decimalType = impalaType.getMinResolutionDecimal(); + toType = factory.createSqlType(SqlTypeName.DECIMAL, + decimalType.decimalPrecision(), decimalType.decimalScale()); + } + return rexBuilder.makeCast(toType, node); + } +} diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedCaseExpr.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedCaseExpr.java new file mode 100644 index 000000000..7aa95aba4 --- /dev/null +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedCaseExpr.java @@ -0,0 +1,69 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.impala.calcite.functions; + +import org.apache.impala.analysis.Analyzer; +import org.apache.impala.analysis.CaseExpr; +import org.apache.impala.analysis.CaseWhenClause; +import org.apache.impala.analysis.Expr; +import org.apache.impala.analysis.FunctionCallExpr; +import org.apache.impala.catalog.Function; +import org.apache.impala.catalog.Type; +import org.apache.impala.common.AnalysisException; + +import java.util.List; + +/** + * A CaseExpr that is always in analyzed state + */ +public class AnalyzedCaseExpr extends CaseExpr { + + private final Function savedFunction_; + private final Type savedType_; + + public AnalyzedCaseExpr(Function fn, List<CaseWhenClause> whenClauses, + Expr elseExpr, Type retType) { + // Calcite never has the first parameter 'case' expression filled in. + super(null, whenClauses, elseExpr); + this.savedFunction_ = fn; + this.savedType_ = retType; + } + + public AnalyzedCaseExpr(Function fn, Type retType, FunctionCallExpr decodeExpr) { + super(decodeExpr); + this.savedFunction_ = fn; + this.savedType_ = retType; + } + + public AnalyzedCaseExpr(AnalyzedCaseExpr other) { + super(other); + this.savedFunction_ = other.savedFunction_; + this.savedType_ = other.savedType_; + } + + @Override + public Expr clone() { + return new AnalyzedCaseExpr(this); + } + + @Override + protected void analyzeImpl(Analyzer analyzer) throws AnalysisException { + this.fn_ = this.savedFunction_; + this.type_ = this.savedType_; + } +} diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedCastExpr.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedCastExpr.java new file mode 100644 index 000000000..b6fa336b6 --- /dev/null +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedCastExpr.java @@ -0,0 +1,55 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.impala.calcite.functions; + +import org.apache.impala.analysis.Analyzer; +import org.apache.impala.analysis.CastExpr; +import org.apache.impala.analysis.Expr; +import org.apache.impala.catalog.Type; +import org.apache.impala.common.AnalysisException; + +/** + * A CastExpr that is always in analyzed state + */ +public class AnalyzedCastExpr extends CastExpr { + + public AnalyzedCastExpr(Type targetType, Expr e) { + super(targetType, e.clone()); + } + + public AnalyzedCastExpr(AnalyzedCastExpr other) { + super(other); + } + + @Override + public Expr clone() { + return new AnalyzedCastExpr(this); + } + + @Override + protected void analyzeImpl(Analyzer analyzer) throws AnalysisException { + } + + /** + * Calcite casts will not be implicit. TODO: need to fix for the toSql routine. + */ + @Override + public boolean isImplicit() { + return false; + } +} diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedFunctionCallExpr.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedFunctionCallExpr.java index 2325cb6e5..c9031ec60 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedFunctionCallExpr.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/AnalyzedFunctionCallExpr.java @@ -78,7 +78,7 @@ public class AnalyzedFunctionCallExpr extends FunctionCallExpr { @Override protected float computeEvalCost() { // TODO: IMPALA-13098: need to implement - return UNKNOWN_COST; + return 1; } @Override diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionResolver.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionResolver.java index e87062303..d2bb9d66e 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionResolver.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/FunctionResolver.java @@ -19,16 +19,22 @@ package org.apache.impala.calcite.functions; import com.google.common.base.Preconditions; import com.google.common.collect.ImmutableMap; +import com.google.common.collect.ImmutableSet; +import com.google.common.collect.Lists; import org.apache.calcite.rel.type.RelDataType; +import org.apache.calcite.rex.RexCall; +import org.apache.calcite.rex.RexNode; import org.apache.calcite.sql.SqlKind; import org.apache.impala.analysis.FunctionName; import org.apache.impala.calcite.type.ImpalaTypeConverter; import org.apache.impala.catalog.BuiltinsDb; import org.apache.impala.catalog.Function; import org.apache.impala.catalog.Type; +import org.apache.impala.catalog.TypeCompatibility; import java.util.List; import java.util.Map; +import java.util.Set; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -43,6 +49,10 @@ public class FunctionResolver { LoggerFactory.getLogger(FunctionResolver.class.getName()); // Map of the Calcite Kind to an Impala function name + // Only functions that do not map directly from Calcite to Impala need + // to be in this map. For instance, Calcite maps "%" to the name "mod" + // and since "mod" is the function name in Impala, it does not need + // to be in this map. public static Map<SqlKind, String> CALCITE_KIND_TO_IMPALA_FUNC = ImmutableMap.<SqlKind, String> builder() .put(SqlKind.EQUALS, "eq") @@ -51,25 +61,77 @@ public class FunctionResolver { .put(SqlKind.LESS_THAN, "lt") .put(SqlKind.LESS_THAN_OR_EQUAL, "le") .put(SqlKind.NOT_EQUALS, "ne") + .put(SqlKind.PLUS, "add") + .put(SqlKind.MINUS, "subtract") + .put(SqlKind.TIMES, "multiply") + .put(SqlKind.DIVIDE, "divide") .build(); - public static Function getFunction(String name, SqlKind kind, + public static Set<SqlKind> ARITHMETIC_TYPES = + ImmutableSet.<SqlKind> builder() + .add(SqlKind.PLUS) + .add(SqlKind.MINUS) + .add(SqlKind.TIMES) + .add(SqlKind.DIVIDE) + .add(SqlKind.MOD) + .build(); + + public static Function getSupertypeFunction(RexCall call) { + // For arithmetic types, we use separate logic. Calcite has already + // calculated the proper return type at validation time. So an + // (INT + INT) function already has the correct calculated return type + // of BIGINT. If we just used the operands, the function resolver would + // return the prototype of INT +(INT, INT) which is not what we want. + // So we pass in the operands based on the already calculated return + // type (call.getType()). + List<RelDataType> argTypes = ARITHMETIC_TYPES.contains(call.getKind()) + ? Lists.newArrayList(call.getType(), call.getType()) + : Lists.transform(call.getOperands(), RexNode::getType); + return getFunction(call.getOperator().getName(), call.getKind(), argTypes, false); + } + + public static Function getSupertypeFunction(String name, SqlKind kind, + List<RelDataType> argTypes) { + return getFunction(name, kind, argTypes, false); + } + + public static Function getExactFunction(String name, SqlKind kind, List<RelDataType> argTypes) { + return getFunction(name, kind, argTypes, true); + } + + public static Function getExactFunction(String name, List<RelDataType> argTypes) { + return getFunction(name, argTypes, true); + } + + public static Function getSupertypeFunction(String name, List<RelDataType> argTypes) { + return getFunction(name, argTypes, false); + } + + private static Function getFunction(String name, SqlKind kind, + List<RelDataType> argTypes, boolean exactMatch) { + // Some names in Calcite don't map exactly to their corresponding Impala + // functions, so we get the right mapping from the HashMap table. String mappedName = CALCITE_KIND_TO_IMPALA_FUNC.get(kind); return mappedName == null - ? getFunction(name, argTypes) - : getFunction(mappedName, argTypes); + ? getFunction(name, argTypes, exactMatch) + : getFunction(mappedName, argTypes, exactMatch); } - public static Function getFunction(String name, List<RelDataType> argTypes) { + private static Function getFunction(String name, List<RelDataType> argTypes, + boolean exactMatch) { String lowercaseName = name.toLowerCase(); - List<Type> impalaArgTypes = ImpalaTypeConverter.getNormalizedImpalaTypes(argTypes); + List<Type> impalaArgTypes = getArgTypes(lowercaseName, argTypes, exactMatch); + Function searchDesc = new Function(new FunctionName(BuiltinsDb.NAME, lowercaseName), impalaArgTypes, Type.INVALID, false); - Function fn = BuiltinsDb.getInstance().getFunction(searchDesc, - Function.CompareMode.IS_INDISTINGUISHABLE); + Function.CompareMode compareMode = exactMatch + ? Function.CompareMode.IS_INDISTINGUISHABLE + : Function.CompareMode.IS_NONSTRICT_SUPERTYPE_OF; + + Function fn = BuiltinsDb.getInstance().getFunction(searchDesc, compareMode); if (fn == null) { LOG.debug("Failed to find function " + lowercaseName); @@ -77,4 +139,52 @@ public class FunctionResolver { return fn; } + + private static List<Type> getArgTypes(String name, List<RelDataType> argTypes, + boolean exactMatch) { + // Case statement is special because the function signature only contains the + // return value as its only operand. This operand needs to be calculated based + // on its parameters (see getCaseArgs method comment) if we are looking for a + // non-exact match (the best case function that would work if casting the operands). + if (name.equals("case")) { + return exactMatch + ? Lists.newArrayList(ImpalaTypeConverter.createImpalaType(argTypes.get(1))) + : getCaseArgs(argTypes); + } + + return ImpalaTypeConverter.getNormalizedImpalaTypes(argTypes); + } + + /** + * getCaseArgs needs to calculate the proper operand. It walks through + * all the "then" clauses (and the default clause) and calculated the + * most compatible type which it will use as its return value. + */ + private static List<Type> getCaseArgs(List<RelDataType> argTypes) { + int numOperands = argTypes.size(); + Type compatibleType = Type.NULL; + for (int i = 0; i < numOperands; ++i) { + if (!shouldSkipOperandForCase(numOperands, i)) { + Type impalaType = ImpalaTypeConverter.createImpalaType(argTypes.get(i)); + compatibleType = Type.getAssignmentCompatibleType(compatibleType, impalaType, + TypeCompatibility.DEFAULT); + } + } + + return Lists.newArrayList(compatibleType); + } + + /** + * shouldSkipOperand checks to see if we should skip this operand from casting. + * Currently it only checks for CASE. CASE will have some boolean operands which + * will never need casting. TODO: support DECODE + */ + public static boolean shouldSkipOperandForCase(int numOperands, int i) { + // The even number parameters are the boolean checks, but the last parameter + // needs to be checked if there is an "else" term. Note: Though there are 2 + // flavors of the CASE command (one with a term before the first WHEN clause), + // Calcite merges these 2 flavors in the validation step which ensures that the + // check here will succeed. + return (i % 2 == 0) && (numOperands - 1 != i); + } } diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/ImplicitTypeChecker.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/ImplicitTypeChecker.java new file mode 100644 index 000000000..e8b8b940e --- /dev/null +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/ImplicitTypeChecker.java @@ -0,0 +1,158 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.impala.calcite.functions; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableSet; +import org.apache.calcite.rel.type.RelDataType; +import org.apache.calcite.sql.type.SqlTypeFamily; +import org.apache.calcite.sql.type.SqlTypeName; +import org.apache.commons.lang3.tuple.Pair; + +import java.util.Set; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Class that checks if an implicit cast is supported through the + * supportsImplicitCasting static method. + */ +public class ImplicitTypeChecker { + protected static final Logger LOG = + LoggerFactory.getLogger(ImplicitTypeChecker.class.getName()); + + // Set containing a pair of casts that are supported. The first type + // is the argument and the second type is the return type. + private static Set<Pair<SqlTypeName, SqlTypeName>> SUPPORTED_IMPLICIT_CASTS = + ImmutableSet.<Pair<SqlTypeName, SqlTypeName>> builder() + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.NULL)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.BOOLEAN)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.TINYINT)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.SMALLINT)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.INTEGER)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.BIGINT)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.FLOAT)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.DECIMAL)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.TIMESTAMP)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.DATE)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.CHAR)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.VARCHAR)) + .add(Pair.of(SqlTypeName.NULL, SqlTypeName.BINARY)) + .add(Pair.of(SqlTypeName.BOOLEAN, SqlTypeName.BOOLEAN)) + .add(Pair.of(SqlTypeName.BOOLEAN, SqlTypeName.TINYINT)) + .add(Pair.of(SqlTypeName.BOOLEAN, SqlTypeName.SMALLINT)) + .add(Pair.of(SqlTypeName.BOOLEAN, SqlTypeName.INTEGER)) + .add(Pair.of(SqlTypeName.BOOLEAN, SqlTypeName.BIGINT)) + .add(Pair.of(SqlTypeName.BOOLEAN, SqlTypeName.FLOAT)) + .add(Pair.of(SqlTypeName.BOOLEAN, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.TINYINT, SqlTypeName.TINYINT)) + .add(Pair.of(SqlTypeName.TINYINT, SqlTypeName.SMALLINT)) + .add(Pair.of(SqlTypeName.TINYINT, SqlTypeName.INTEGER)) + .add(Pair.of(SqlTypeName.TINYINT, SqlTypeName.BIGINT)) + .add(Pair.of(SqlTypeName.TINYINT, SqlTypeName.FLOAT)) + .add(Pair.of(SqlTypeName.TINYINT, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.TINYINT, SqlTypeName.DECIMAL)) + .add(Pair.of(SqlTypeName.SMALLINT, SqlTypeName.SMALLINT)) + .add(Pair.of(SqlTypeName.SMALLINT, SqlTypeName.INTEGER)) + .add(Pair.of(SqlTypeName.SMALLINT, SqlTypeName.BIGINT)) + .add(Pair.of(SqlTypeName.SMALLINT, SqlTypeName.FLOAT)) + .add(Pair.of(SqlTypeName.SMALLINT, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.SMALLINT, SqlTypeName.DECIMAL)) + .add(Pair.of(SqlTypeName.INTEGER, SqlTypeName.INTEGER)) + .add(Pair.of(SqlTypeName.INTEGER, SqlTypeName.BIGINT)) + .add(Pair.of(SqlTypeName.INTEGER, SqlTypeName.FLOAT)) + .add(Pair.of(SqlTypeName.INTEGER, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.INTEGER, SqlTypeName.DECIMAL)) + .add(Pair.of(SqlTypeName.BIGINT, SqlTypeName.BIGINT)) + .add(Pair.of(SqlTypeName.BIGINT, SqlTypeName.FLOAT)) + .add(Pair.of(SqlTypeName.BIGINT, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.BIGINT, SqlTypeName.DECIMAL)) + .add(Pair.of(SqlTypeName.FLOAT, SqlTypeName.FLOAT)) + .add(Pair.of(SqlTypeName.FLOAT, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.DOUBLE, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.TIMESTAMP, SqlTypeName.TIMESTAMP)) + .add(Pair.of(SqlTypeName.CHAR, SqlTypeName.CHAR)) + .add(Pair.of(SqlTypeName.CHAR, SqlTypeName.VARCHAR)) + .add(Pair.of(SqlTypeName.DATE, SqlTypeName.DATE)) + .add(Pair.of(SqlTypeName.DECIMAL, SqlTypeName.DECIMAL)) + .add(Pair.of(SqlTypeName.DECIMAL, SqlTypeName.FLOAT)) + .add(Pair.of(SqlTypeName.DECIMAL, SqlTypeName.DOUBLE)) + .add(Pair.of(SqlTypeName.DATE, SqlTypeName.TIMESTAMP)) + .add(Pair.of(SqlTypeName.BINARY, SqlTypeName.BINARY)) + // These casts aren't compatible within Impala, but they are used in + // arithmetic operations like "float_col + decimal_col", so Calcite does + // need these supported. + .add(Pair.of(SqlTypeName.FLOAT, SqlTypeName.DECIMAL)) + .add(Pair.of(SqlTypeName.DOUBLE, SqlTypeName.DECIMAL)) + .build(); + + // Check to see if the "from" type can be cast up to the "to" type. In the case + // where both are decimals, this will return true. + public static boolean supportsImplicitCasting(RelDataType castFrom, + RelDataType castTo) { + + if (SUPPORTED_IMPLICIT_CASTS.contains( + Pair.of(castFrom.getSqlTypeName(), castTo.getSqlTypeName()))) { + return true; + } + + if (areCompatibleDataTypes(castFrom, castTo)) { + return true; + } + + return false; + } + + private static boolean areCompatibleDataTypes(RelDataType fromType, + RelDataType toType) { + + // special cases for string. The varchar type is overloaded for both varchar + // and string types. + if (fromType.getSqlTypeName().equals(SqlTypeName.VARCHAR) && + fromType.getPrecision() == Integer.MAX_VALUE) { + // string to string is ok. string to varchar is not. + if (toType.getPrecision() == Integer.MAX_VALUE) { + return true; + } + switch (toType.getSqlTypeName()) { + case DATE: + case TIMESTAMP: + return true; + default: + return false; + } + } + + // Calcite has interval types where Impala needs some kind of numeric type, so + // these are compatible. + if ((fromType.getSqlTypeName().getFamily() == SqlTypeFamily.INTERVAL_DAY_TIME) || + (fromType.getSqlTypeName().getFamily() == SqlTypeFamily.INTERVAL_YEAR_MONTH) || + (toType.getSqlTypeName().getFamily() == SqlTypeFamily.INTERVAL_DAY_TIME) || + (toType.getSqlTypeName().getFamily() == SqlTypeFamily.INTERVAL_YEAR_MONTH)) { + if ((fromType.getSqlTypeName().getFamily() == SqlTypeFamily.NUMERIC) || + (toType.getSqlTypeName().getFamily() == SqlTypeFamily.NUMERIC)) { + return true; + } + } + + return fromType.getSqlTypeName() == toType.getSqlTypeName(); + } + +} diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexCallConverter.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexCallConverter.java index 11f513d11..468abec5b 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexCallConverter.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexCallConverter.java @@ -26,6 +26,8 @@ import org.apache.calcite.rex.RexCall; import org.apache.calcite.rex.RexNode; import org.apache.calcite.sql.SqlKind; import org.apache.impala.analysis.BinaryPredicate; +import org.apache.impala.analysis.CaseWhenClause; +import org.apache.impala.analysis.CompoundPredicate; import org.apache.impala.analysis.Expr; import org.apache.impala.calcite.type.ImpalaTypeConverter; import org.apache.impala.catalog.Function; @@ -33,7 +35,7 @@ import org.apache.impala.catalog.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; - +import java.util.ArrayList; import java.util.List; import java.util.Map; @@ -55,15 +57,34 @@ public class RexCallConverter { .put(SqlKind.IS_DISTINCT_FROM, BinaryPredicate.Operator.DISTINCT_FROM) .put(SqlKind.IS_NOT_DISTINCT_FROM, BinaryPredicate.Operator.NOT_DISTINCT) .build(); + /* * Returns the Impala Expr object for RexCallConverter. */ public static Expr getExpr(RexCall rexCall, List<Expr> params, RexBuilder rexBuilder) { + // Some functions are known just based on their RexCall signature. + switch (rexCall.getOperator().getKind()) { + case OR: + case AND: + return createCompoundExpr(rexCall, params); + case CAST: + return createCastExpr(rexCall, params); + } + String funcName = rexCall.getOperator().getName().toLowerCase(); Function fn = getFunction(rexCall); + if (fn == null) { + List<RelDataType> argTypes = + Lists.transform(rexCall.getOperands(), RexNode::getType); + Preconditions.checkState(false, "Could not find function \"" + funcName + + "\" in Impala " + "with args " + argTypes + " and return type " + + rexCall.getType()); + return null; + } + Type impalaRetType = ImpalaTypeConverter.createImpalaType(fn.getReturnType(), rexCall.getType().getPrecision(), rexCall.getType().getScale()); @@ -72,18 +93,65 @@ public class RexCallConverter { impalaRetType); } - return new AnalyzedFunctionCallExpr(fn, params, impalaRetType); + switch (rexCall.getOperator().getKind()) { + case CASE: + return createCaseExpr(fn, params, impalaRetType); + default: + return new AnalyzedFunctionCallExpr(fn, params, impalaRetType); + } } private static Function getFunction(RexCall call) { List<RelDataType> argTypes = Lists.transform(call.getOperands(), RexNode::getType); String name = call.getOperator().getName(); - Function fn = FunctionResolver.getFunction(name, call.getKind(), argTypes); - Preconditions.checkNotNull(fn, "Could not find function \"" + name + "\" in Impala " - + "with args " + argTypes + " and return type " + call.getType()); - return fn; + return FunctionResolver.getExactFunction(name, call.getKind(), argTypes); + } + + /** + * Create a Compound Expr + */ + private static Expr createCompoundExpr(RexCall rexCall, List<Expr> params) { + switch (rexCall.getOperator().getKind()) { + case OR: + return CompoundPredicate.createDisjunctivePredicate(params); + case AND: + return CompoundPredicate.createConjunctivePredicate(params); + } + Preconditions.checkState(false, "Unknown type: " + rexCall.getOperator().getKind()); + return null; } + private static Expr createCastExpr(RexCall call, List<Expr> params) { + Type impalaRetType = ImpalaTypeConverter.createImpalaType(call.getType()); + if (params.get(0).getType() == Type.NULL) { + return new AnalyzedNullLiteral(impalaRetType); + } + + // no need for redundant cast. + if (params.get(0).getType().equals(impalaRetType)) { + return params.get(0); + } + + return new AnalyzedCastExpr(impalaRetType, params.get(0)); + } + + private static Expr createCaseExpr(Function fn, List<Expr> params, Type retType) { + List<CaseWhenClause> caseWhenClauses = new ArrayList<>(); + Expr whenParam = null; + // params alternate between "when" and the action expr + for (Expr param : params) { + if (whenParam == null) { + whenParam = param; + } else { + caseWhenClauses.add(new CaseWhenClause(whenParam, param)); + whenParam = null; + } + } + // Leftover 'when' param is the 'else' param, null if there is no leftover + return new AnalyzedCaseExpr(fn, caseWhenClauses, whenParam, retType); + } + + private static Expr createBinaryCompExpr(Function fn, List<Expr> params, SqlKind sqlKind, Type retType) { Preconditions.checkArgument(params.size() == 2); diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexLiteralConverter.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexLiteralConverter.java index 626b2af8b..b1a03c2bd 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexLiteralConverter.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/functions/RexLiteralConverter.java @@ -74,10 +74,6 @@ public class RexLiteralConverter { ScalarType charType = rexLiteral.getType().getSqlTypeName() == SqlTypeName.VARCHAR ? Type.STRING : ScalarType.createCharType(rexLiteral.getValueAs(String.class).length()); - if (charType.getPrimitiveType() == PrimitiveType.CHAR) { - // ensure no wildcards or char length 0 which will crash impalad - Preconditions.checkState(charType.getLength() > 0); - } Expr charExpr = new StringLiteral(rexLiteral.getValueAs(String.class), charType, false); return charExpr; @@ -109,7 +105,7 @@ public class RexLiteralConverter { String timestamp = rexLiteral.getValueAs(TimestampString.class).toString(); List<Expr> argList = Lists.newArrayList(new StringLiteral(timestamp, Type.STRING, false)); - Function castFunc = FunctionResolver.getFunction("casttotimestamp", typeNames); + Function castFunc = FunctionResolver.getExactFunction("casttotimestamp", typeNames); return new AnalyzedFunctionCallExpr(castFunc, argList, Type.TIMESTAMP); } } diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaConvertletTable.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaConvertletTable.java new file mode 100644 index 000000000..535578990 --- /dev/null +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaConvertletTable.java @@ -0,0 +1,50 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.impala.calcite.operators; + +import org.apache.calcite.sql.SqlCall; +import org.apache.calcite.sql.fun.SqlStdOperatorTable; +import org.apache.calcite.sql2rel.ReflectiveConvertletTable; +import org.apache.calcite.sql2rel.SqlRexConvertlet; +import org.apache.calcite.sql2rel.StandardConvertletTable; +import org.apache.impala.calcite.operators.ImpalaCustomOperatorTable; + +/** + * + */ +public class ImpalaConvertletTable extends ReflectiveConvertletTable { + public static final ImpalaConvertletTable INSTANCE = + new ImpalaConvertletTable(); + + public ImpalaConvertletTable() { + addAlias(ImpalaCustomOperatorTable.PERCENT_REMAINDER, SqlStdOperatorTable.MOD); + } + + @Override + public SqlRexConvertlet get(SqlCall call) { + // If we were using Calcite's PERCENT_REMAINDER, the "addAlias" is defined + // in StandardConvertletTable. But since the PERCENT_REMAINDER is overridden + // with an Impala version (which derives the Impala return type), we need + // to handle the alias in our own convertlet. + if (call.getOperator().equals(ImpalaCustomOperatorTable.PERCENT_REMAINDER)) { + return super.get(call); + } + + return StandardConvertletTable.INSTANCE.get(call); + } + +} diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaCustomOperatorTable.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaCustomOperatorTable.java new file mode 100644 index 000000000..db29c7c54 --- /dev/null +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaCustomOperatorTable.java @@ -0,0 +1,201 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.impala.calcite.operators; + +import org.apache.calcite.rel.type.RelDataType; +import org.apache.calcite.rel.type.RelDataTypeFactory; +import org.apache.calcite.sql.ExplicitOperatorBinding; +import org.apache.calcite.sql.SqlBinaryOperator; +import org.apache.calcite.sql.SqlKind; +import org.apache.calcite.sql.SqlOperatorBinding; +import org.apache.calcite.sql.fun.SqlMonotonicBinaryOperator; +import org.apache.calcite.sql.type.InferTypes; +import org.apache.calcite.sql.type.OperandTypes; +import org.apache.calcite.sql.type.SqlReturnTypeInference; +import org.apache.calcite.sql.type.SqlTypeName; +import org.apache.calcite.sql.type.SqlTypeUtil; +import org.apache.calcite.sql.type.SqlTypeTransforms; +import org.apache.calcite.sql.util.ReflectiveSqlOperatorTable; +import org.apache.impala.analysis.ArithmeticExpr; +import org.apache.impala.calcite.type.ImpalaTypeConverter; +import org.apache.impala.calcite.type.ImpalaTypeSystemImpl; +import org.apache.impala.catalog.Type; + +import com.google.common.base.Suppliers; +import com.google.common.collect.ImmutableList; + +import java.math.BigDecimal; +import java.util.List; +import java.util.function.Supplier; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class ImpalaCustomOperatorTable extends ReflectiveSqlOperatorTable { + + protected static final Logger LOG = LoggerFactory.getLogger( + ImpalaCustomOperatorTable.class.getName()); + //~ Static fields/initializers --------------------------------------------- + + private static final RelDataType inferReturnTypeForArithmeticOps( + SqlOperatorBinding opBinding, ArithmeticExpr.Operator op) { + + List<RelDataType> operandTypes = opBinding.collectOperandTypes(); + + if (SqlTypeUtil.isDate(operandTypes.get(0)) || + SqlTypeUtil.isDate(operandTypes.get(1))) { + return ImpalaTypeConverter.getRelDataType(Type.DATE); + } + + if (SqlTypeUtil.isDatetime(operandTypes.get(0)) || + SqlTypeUtil.isDatetime(operandTypes.get(1))) { + return ImpalaTypeConverter.getRelDataType(Type.TIMESTAMP); + } + + RelDataType type0 = + getOperandForArithmeticOpSqlOperatorBinding(opBinding, operandTypes, 0); + RelDataType type1 = + getOperandForArithmeticOpSqlOperatorBinding(opBinding, operandTypes, 1); + + RelDataTypeFactory typeFactory = opBinding.getTypeFactory(); + ImpalaTypeSystemImpl typeSystemImpl = + (ImpalaTypeSystemImpl) typeFactory.getTypeSystem(); + return typeSystemImpl.deriveArithmeticType(typeFactory, type0, type1, op); + }; + + private static RelDataType getOperandForArithmeticOpSqlOperatorBinding( + SqlOperatorBinding opBinding, List<RelDataType> operandTypes, int index) { + if (opBinding instanceof ExplicitOperatorBinding) { + return operandTypes.get(index); + } else if (opBinding.isOperandNull(index, true)) { + // for nulltypes, just assume smallest possible number type + return ImpalaTypeConverter.getRelDataType(Type.TINYINT); + } else if (operandTypes.get(index).getSqlTypeName().equals(SqlTypeName.INTEGER) && + opBinding.isOperandLiteral(index, true)) { + // For literal types, currently Calcite treats all of them as INTEGERs. Impala + // requires the smallest possible type (e.g. 2 should be a TINYINT), and needs + // this to infer the proper return type. Note though: this method is only used + // for inferring the return type and not coercing the operand, which will stay + // an INTEGER for now. The operand will be coerced later in the compilation, + // under the coercenodes modules. + BigDecimal bd0 = opBinding.getOperandLiteralValue(index, BigDecimal.class); + RelDataType rdt = + ImpalaTypeConverter.getLiteralDataType(bd0, opBinding.getOperandType(index)); + return rdt; + } + + return opBinding.getOperandType(index); + } + + private static final Supplier<ImpalaCustomOperatorTable> INSTANCE = + Suppliers.memoize(() -> + (ImpalaCustomOperatorTable) new ImpalaCustomOperatorTable().init()); + + public static final SqlReturnTypeInference ADD_ADJUSTED_RETURN_TYPE = opBinding -> { + return inferReturnTypeForArithmeticOps(opBinding, ArithmeticExpr.Operator.ADD); + }; + + public static final SqlReturnTypeInference ADD_ADJUSTED_RETURN_TYPE_NULLABLE = + ADD_ADJUSTED_RETURN_TYPE.andThen(SqlTypeTransforms.TO_NULLABLE); + + public static final SqlReturnTypeInference MINUS_ADJUSTED_RETURN_TYPE = opBinding -> { + return inferReturnTypeForArithmeticOps(opBinding, ArithmeticExpr.Operator.SUBTRACT); + }; + + public static final SqlReturnTypeInference MINUS_ADJUSTED_RETURN_TYPE_NULLABLE = + MINUS_ADJUSTED_RETURN_TYPE.andThen(SqlTypeTransforms.TO_NULLABLE); + + public static final SqlReturnTypeInference MULT_ADJUSTED_RETURN_TYPE = opBinding -> { + return inferReturnTypeForArithmeticOps(opBinding, ArithmeticExpr.Operator.MULTIPLY); + }; + + public static final SqlReturnTypeInference MULT_ADJUSTED_RETURN_TYPE_NULLABLE = + MULT_ADJUSTED_RETURN_TYPE.andThen(SqlTypeTransforms.TO_NULLABLE); + + public static final SqlReturnTypeInference DIVIDE_ADJUSTED_RETURN_TYPE = opBinding -> { + return inferReturnTypeForArithmeticOps(opBinding, ArithmeticExpr.Operator.DIVIDE); + }; + + public static final SqlReturnTypeInference DIVIDE_ADJUSTED_RETURN_TYPE_NULLABLE = + DIVIDE_ADJUSTED_RETURN_TYPE.andThen(SqlTypeTransforms.TO_NULLABLE); + + public static final SqlReturnTypeInference MOD_ADJUSTED_RETURN_TYPE = opBinding -> { + return inferReturnTypeForArithmeticOps(opBinding, ArithmeticExpr.Operator.MOD); + }; + + public static final SqlReturnTypeInference MOD_ADJUSTED_RETURN_TYPE_NULLABLE = + MOD_ADJUSTED_RETURN_TYPE.andThen(SqlTypeTransforms.TO_NULLABLE); + + public static final SqlReturnTypeInference STRING_TYPE = opBinding -> { + return ImpalaTypeConverter.getRelDataType(Type.STRING); + }; + + public static final SqlBinaryOperator PLUS = + new SqlMonotonicBinaryOperator( + "+", + SqlKind.PLUS, + 40, + true, + ADD_ADJUSTED_RETURN_TYPE_NULLABLE, + InferTypes.FIRST_KNOWN, + OperandTypes.PLUS_OPERATOR); + + public static final SqlBinaryOperator MINUS = + new SqlMonotonicBinaryOperator( + "-", + SqlKind.MINUS, + 40, + true, + MINUS_ADJUSTED_RETURN_TYPE_NULLABLE, + InferTypes.FIRST_KNOWN, + OperandTypes.MINUS_OPERATOR); + + public static final SqlBinaryOperator MULTIPLY = + new SqlMonotonicBinaryOperator( + "*", + SqlKind.TIMES, + 60, + true, + MULT_ADJUSTED_RETURN_TYPE_NULLABLE, + InferTypes.FIRST_KNOWN, + OperandTypes.MULTIPLY_OPERATOR); + + public static final SqlBinaryOperator DIVIDE = + new SqlBinaryOperator( + "/", + SqlKind.DIVIDE, + 60, + true, + DIVIDE_ADJUSTED_RETURN_TYPE_NULLABLE, + InferTypes.FIRST_KNOWN, + OperandTypes.DIVISION_OPERATOR); + + public static final SqlBinaryOperator PERCENT_REMAINDER = + new SqlBinaryOperator( + "%", + SqlKind.MOD, + 60, + true, + MOD_ADJUSTED_RETURN_TYPE_NULLABLE, + null, + OperandTypes.NUMERIC_NUMERIC); + + public static ImpalaCustomOperatorTable instance() { + return INSTANCE.get(); + } +} diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperator.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperator.java index 1024d7818..538d32ee4 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperator.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperator.java @@ -19,7 +19,6 @@ package org.apache.impala.calcite.operators; import com.google.common.base.Preconditions; -import org.apache.commons.lang.StringUtils; import org.apache.impala.calcite.type.ImpalaTypeSystemImpl; import org.apache.calcite.jdbc.JavaTypeFactoryImpl; import org.apache.calcite.rel.type.RelDataType; @@ -33,18 +32,13 @@ import org.apache.calcite.sql.SqlOperandCountRange; import org.apache.calcite.sql.SqlOperatorBinding; import org.apache.calcite.sql.SqlSyntax; import org.apache.calcite.sql.type.SqlOperandCountRanges; -import org.apache.impala.analysis.FunctionName; import org.apache.impala.calcite.functions.FunctionResolver; import org.apache.impala.calcite.type.ImpalaTypeConverter; -import org.apache.impala.catalog.BuiltinsDb; import org.apache.impala.catalog.Function; import org.apache.impala.catalog.Type; import java.util.ArrayList; -import java.util.Collections; -import java.util.LinkedHashMap; import java.util.List; -import java.util.Map; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -77,9 +71,7 @@ public class ImpalaOperator extends SqlFunction { RelDataTypeFactory factory = rexBuilder.getTypeFactory(); // Resolve Impala function through Impala method. - // TODO: IMPALA-13022: Right now, CompareMode is INDISTINGUISHABLE because this - // commit only deals with exact matches. This will change in a future commit. - Function fn = FunctionResolver.getFunction(getName(), operandTypes); + Function fn = FunctionResolver.getSupertypeFunction(getName(), operandTypes); if (fn == null) { throw new IllegalArgumentException("Cannot infer return type for " diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java index 6e2540e94..7aee0ab37 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/operators/ImpalaOperatorTable.java @@ -25,15 +25,11 @@ import org.apache.calcite.sql.SqlOperator; import org.apache.calcite.sql.SqlSyntax; import org.apache.calcite.sql.fun.SqlStdOperatorTable; import org.apache.calcite.sql.util.ReflectiveSqlOperatorTable; -import org.apache.impala.calcite.functions.FunctionResolver; import org.apache.impala.catalog.BuiltinsDb; import org.apache.impala.catalog.Db; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; + import java.util.List; -import java.util.Map; -import java.util.Set; + import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -67,6 +63,13 @@ public class ImpalaOperatorTable extends ReflectiveSqlOperatorTable { SqlSyntax syntax, List<SqlOperator> operatorList, SqlNameMatcher nameMatcher) { + ImpalaCustomOperatorTable.instance().lookupOperatorOverloads(opName, category, syntax, + operatorList, nameMatcher); + + if (operatorList.size() == 1) { + return; + } + // Check Calcite operator table for existence. SqlStdOperatorTable.instance().lookupOperatorOverloads(opName, category, syntax, operatorList, nameMatcher); diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaAggRel.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaAggRel.java index 3f4e613a8..2805d0267 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaAggRel.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaAggRel.java @@ -299,7 +299,7 @@ public class ImpalaAggRel extends Aggregate RelDataType relDataType = input.getRowType().getFieldList().get(i).getType(); operandTypes.add(relDataType); } - return FunctionResolver.getFunction(aggFunction.getName(), aggFunction.getKind(), + return FunctionResolver.getExactFunction(aggFunction.getName(), aggFunction.getKind(), operandTypes); } diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaJoinRel.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaJoinRel.java index 4c90c9bc7..bcaf7241b 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaJoinRel.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaJoinRel.java @@ -233,7 +233,7 @@ public class ImpalaJoinRel extends Join tmpArgs.add(nullLiteral); tmpArgs.add(expr); List<Type> typeNames = ImmutableList.of(Type.BOOLEAN, expr.getType(), expr.getType()); - Function conditionalFunc = FunctionResolver.getFunction("if", + Function conditionalFunc = FunctionResolver.getExactFunction("if", ImpalaTypeConverter.getRelDataTypesForArgs(typeNames)); Preconditions.checkNotNull(conditionalFunc, "Could not create IF function for arg types %s and return type %s", diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java index 065117805..6500f927f 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaPlanRel.java @@ -17,6 +17,15 @@ package org.apache.impala.calcite.rel.node; +import org.apache.calcite.rel.RelNode; +import org.apache.calcite.rel.core.Aggregate; +import org.apache.calcite.rel.core.Filter; +import org.apache.calcite.rel.core.Join; +import org.apache.calcite.rel.core.Project; +import org.apache.calcite.rel.core.Sort; +import org.apache.calcite.rel.core.TableScan; +import org.apache.calcite.rel.core.Union; +import org.apache.calcite.rel.core.Values; import org.apache.impala.common.ImpalaException; /** @@ -53,4 +62,32 @@ public interface ImpalaPlanRel { * Return the RelNodeType enum of the implementing class of ImpalaPlanRel. */ public RelNodeType relNodeType(); + + public static RelNodeType getRelNodeType(RelNode relNode) { + if (relNode instanceof Aggregate) { + return RelNodeType.AGGREGATE; + } + if (relNode instanceof Filter) { + return RelNodeType.FILTER; + } + if (relNode instanceof TableScan) { + return RelNodeType.HDFSSCAN; + } + if (relNode instanceof Join) { + return RelNodeType.JOIN; + } + if (relNode instanceof Project) { + return RelNodeType.PROJECT; + } + if (relNode instanceof Sort) { + return RelNodeType.SORT; + } + if (relNode instanceof Union) { + return RelNodeType.UNION; + } + if (relNode instanceof Values) { + return RelNodeType.VALUES; + } + throw new RuntimeException("Unknown RelNode: " + relNode); + } } diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaValuesRel.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaValuesRel.java index 6a39eb573..51b3953e6 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaValuesRel.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaValuesRel.java @@ -20,11 +20,11 @@ import com.google.common.base.Preconditions; import org.apache.calcite.rel.core.Values; import org.apache.calcite.rel.type.RelDataType; import org.apache.calcite.rex.RexLiteral; -import org.apache.calcite.sql.type.SqlTypeName; import org.apache.impala.analysis.Expr; +import org.apache.impala.analysis.LiteralExpr; import org.apache.impala.calcite.rel.util.ExprConjunctsConverter; -import org.apache.impala.common.AnalysisException; +import org.apache.impala.calcite.type.ImpalaTypeConverter; import org.apache.impala.common.ImpalaException; import org.apache.impala.planner.PlanNode; import org.apache.impala.planner.PlanNodeId; @@ -32,6 +32,8 @@ import org.apache.impala.planner.PlanNodeId; import java.util.ArrayList; import java.util.List; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; /** * ImpalaValuesRel handles the Values RelNode in Calcite. * @@ -46,6 +48,8 @@ import java.util.List; public class ImpalaValuesRel extends Values implements ImpalaPlanRel { + protected static final Logger LOG = + LoggerFactory.getLogger(ImpalaValuesRel.class.getName()); public ImpalaValuesRel(Values values) { super(values.getCluster(), values.getRowType(), values.getTuples(), values.getTraitSet()); @@ -61,7 +65,8 @@ public class ImpalaValuesRel extends Values PlanNodeId nodeId = context.ctx_.getNextNodeId(); - RelDataType rowType = getRowType(); + RelDataType rowType = + context.parentRowType_ != null ? context.parentRowType_ : getRowType(); List<NodeWithExprs> nodeWithExprsList = getValuesExprs(context); @@ -89,24 +94,22 @@ public class ImpalaValuesRel extends Values PlanNode retNode = null; List<Expr> outputExprs = new ArrayList<>(); + int i = 0; for (RexLiteral literal : literals) { - // TODO: IMPALA-13022: Char types are currently disabled. The problem is a bit - // complex. Impala expects string literals to be of type STRING. Calcite does not - // have a type STRING and instead creates literals of type CHAR<x>, where <x> is - // the size of the char literal. This causes a couple of problems: - // 1) If there is a Union on top of a Values (e.g. select 'hello' union select - // 'goodbye') there will be a type mismatch (e.g. char(5) and char(7)) which will - // cause an impalad crash. The server crash is the main reason for disabling this - // 2) The return type of the root expression should be "string". While this really - // only will matter once CTAS support is enabled, it still is something that should - // be flagged as not working right now. - if (literal.getType().getSqlTypeName().equals(SqlTypeName.CHAR)) { - throw new AnalysisException("Char type values are not yet supported."); - } ExprConjunctsConverter converter = new ExprConjunctsConverter(literal, new ArrayList<>(), getCluster().getRexBuilder(), context.ctx_.getRootAnalyzer()); - outputExprs.addAll(converter.getImpalaConjuncts()); + + if (context.parentRowType_ != null) { + LiteralExpr e = (LiteralExpr) converter.getImpalaConjuncts().get(0); + LiteralExpr f = LiteralExpr.createFromUnescapedStr(e.getStringValue(), + ImpalaTypeConverter.createImpalaType( + context.parentRowType_.getFieldList().get(i).getType())); + outputExprs.add(f); + } else { + outputExprs.addAll(converter.getImpalaConjuncts()); + } + i++; } return new NodeWithExprs(retNode, outputExprs); diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java index 50953665f..aca145c8b 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ParentPlanRelContext.java @@ -17,6 +17,7 @@ package org.apache.impala.calcite.rel.node; +import org.apache.calcite.rel.type.RelDataType; import org.apache.calcite.rex.RexNode; import org.apache.calcite.util.ImmutableBitSet; import org.apache.impala.planner.PlannerContext; @@ -40,6 +41,8 @@ public class ParentPlanRelContext { // type of parent RelNode public final ImpalaPlanRel.RelNodeType parentType_; + public final RelDataType parentRowType_; + /** * Constructor meant for root node. */ @@ -48,6 +51,7 @@ public class ParentPlanRelContext { this.filterCondition_ = null; this.inputRefs_ = null; this.parentType_ = null; + this.parentRowType_ = null; } private ParentPlanRelContext(Builder builder) { @@ -55,6 +59,7 @@ public class ParentPlanRelContext { this.filterCondition_ = builder.filterCondition_; this.inputRefs_ = builder.inputRefs_; this.parentType_ = builder.parentType_; + this.parentRowType_ = builder.parentRowType_; } public static class Builder { @@ -62,6 +67,7 @@ public class ParentPlanRelContext { private RexNode filterCondition_; private ImmutableBitSet inputRefs_; private ImpalaPlanRel.RelNodeType parentType_; + private RelDataType parentRowType_; /** * Should only be called from root level. @@ -85,6 +91,14 @@ public class ParentPlanRelContext { this.inputRefs_ = inputRefs; } + public void setParentRowType(RelDataType parentRowType) { + this.parentRowType_ = parentRowType; + } + + public void setParentType(ImpalaPlanRel.RelNodeType parentType) { + this.parentType_ = parentType; + } + public ParentPlanRelContext build() { return new ParentPlanRelContext(this); } diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java index f42780430..60f4d89fe 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java @@ -30,6 +30,7 @@ import org.apache.calcite.rel.rules.CoreRules; import org.apache.calcite.sql.SqlExplainFormat; import org.apache.calcite.sql.SqlExplainLevel; import org.apache.calcite.tools.RelBuilder; +import org.apache.impala.calcite.coercenodes.CoerceNodes; import org.apache.impala.calcite.rel.node.ConvertToImpalaRelRules; import org.apache.impala.calcite.rel.node.ImpalaPlanRel; import org.apache.impala.common.ImpalaException; @@ -101,6 +102,9 @@ public class CalciteOptimizer implements CompilerStep { public ImpalaPlanRel runImpalaConvertProgram(RelBuilder relBuilder, RelNode plan) throws ImpalaException { + RelNode preConversionNode = + CoerceNodes.coerceNodes(plan, plan.getCluster().getRexBuilder()); + HepProgramBuilder builder = new HepProgramBuilder(); builder.addMatchOrder(HepMatchOrder.BOTTOM_UP); @@ -115,7 +119,7 @@ public class CalciteOptimizer implements CompilerStep { new ConvertToImpalaRelRules.ImpalaValuesRule() )); - return (ImpalaPlanRel) runProgram(plan, builder.build()); + return (ImpalaPlanRel) runProgram(preConversionNode, builder.build()); } private RelNode runProgram(RelNode currentNode, HepProgram program) { @@ -138,6 +142,6 @@ public class CalciteOptimizer implements CompilerStep { LOG.debug("Finished optimizer step, but unknown result: " + resultObject); return; } - LOG.debug(getDebugString(resultObject)); + LOG.info(getDebugString(resultObject)); } } diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java index 7f10c51d3..30ce7e444 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java @@ -30,7 +30,7 @@ import org.apache.calcite.sql.SqlExplainFormat; import org.apache.calcite.sql.SqlExplainLevel; import org.apache.calcite.sql.SqlNode; import org.apache.calcite.sql2rel.SqlToRelConverter; -import org.apache.calcite.sql2rel.StandardConvertletTable; +import org.apache.impala.calcite.operators.ImpalaConvertletTable; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -66,7 +66,7 @@ public class CalciteRelNodeConverter implements CompilerStep { validator_.getSqlValidator(), validator_.getCatalogReader(), cluster_, - StandardConvertletTable.INSTANCE, + ImpalaConvertletTable.INSTANCE, SqlToRelConverter.config()); // Convert the valid AST into a logical plan diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java index dc559a0fc..978f49b68 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeConverter.java @@ -17,6 +17,8 @@ package org.apache.impala.calcite.type; +import com.google.common.base.Preconditions; +import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Lists; import org.apache.calcite.jdbc.JavaTypeFactoryImpl; @@ -24,10 +26,13 @@ import org.apache.calcite.rel.type.RelDataType; import org.apache.calcite.rel.type.RelDataTypeFactory; import org.apache.calcite.rex.RexBuilder; import org.apache.calcite.sql.type.SqlTypeName; +import org.apache.impala.analysis.NumericLiteral; import org.apache.impala.catalog.ScalarType; import org.apache.impala.catalog.Type; +import org.apache.impala.catalog.TypeCompatibility; import org.apache.impala.thrift.TPrimitiveType; +import java.math.BigDecimal; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -341,4 +346,62 @@ public class ImpalaTypeConverter { public static List<RelDataType> getRelDataTypesForArgs(List<Type> impalaTypes) { return Lists.transform(impalaTypes, ImpalaTypeConverter::getRelDataType); } + + // Converts Calcite Integer literal type into an appropriate exact type for Impala, + // e.g. TINYINT, SMALLINT, INT, or BIGINT + public static RelDataType getLiteralDataType(BigDecimal bd, RelDataType rdt) { + RexBuilder rexBuilder = + new RexBuilder(new JavaTypeFactoryImpl(new ImpalaTypeSystemImpl())); + RelDataTypeFactory factory = rexBuilder.getTypeFactory(); + + // If value is null, just use smallest value + if (bd == null) { + return getRelDataType(Type.TINYINT); + } + + // If it has a scale, it has to be a decimal, so leave it as a decimal. + if (bd.scale() > 0) { + return rdt; + } + + if (NumericLiteral.fitsInTinyInt(bd)) { + rdt = getRelDataType(Type.TINYINT); + } else if (NumericLiteral.fitsInSmallInt(bd)) { + rdt = getRelDataType(Type.SMALLINT); + } else if (NumericLiteral.fitsInInt(bd)) { + rdt = getRelDataType(Type.INT); + } else { + rdt = getRelDataType(Type.BIGINT); + } + return factory.createTypeWithNullability(rdt, false); + } + + public static RelDataType getCompatibleType(List<RelDataType> dataTypes, + RelDataTypeFactory factory) { + Preconditions.checkState(dataTypes.size() > 0); + if (dataTypes.size() == 1) { + return dataTypes.get(0); + } + RelDataType commonType = dataTypes.get(0); + for (int i = 1; i < dataTypes.size(); ++i) { + commonType = getCompatibleType(commonType, dataTypes.get(i), factory); + } + return commonType; + } + + public static RelDataType getCompatibleType( + RelDataType type1, RelDataType type2, RelDataTypeFactory factory) { + // can't handle nulls, but let caller handle this. + if (type1 == null || type2 == null) { + return null; + } + + Type impalaType1 = createImpalaType(type1); + Type impalaType2 = createImpalaType(type2); + + Type retType = ScalarType.getAssignmentCompatibleType(impalaType1, impalaType2, + TypeCompatibility.DEFAULT); + + return createRelDataType(factory, retType); + } } diff --git a/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java b/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java index c812b461f..799d89e75 100644 --- a/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java +++ b/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java @@ -20,9 +20,7 @@ package org.apache.impala.calcite.type; import org.apache.calcite.rel.type.RelDataType; import org.apache.calcite.rel.type.RelDataTypeFactory; import org.apache.calcite.rel.type.RelDataTypeSystemImpl; -import org.apache.calcite.sql.SqlKind; import org.apache.calcite.sql.type.SqlTypeName; -import org.apache.calcite.sql.type.SqlTypeUtil; import org.apache.impala.analysis.ArithmeticExpr; import org.apache.impala.analysis.TypesUtil; import org.apache.impala.catalog.ScalarType; @@ -30,7 +28,6 @@ import org.apache.impala.catalog.Type; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.apache.commons.lang3.exception.ExceptionUtils; /** * ImpalaTypeSystemImpl contains constants that are specific @@ -182,4 +179,115 @@ public class ImpalaTypeSystemImpl extends RelDataTypeSystemImpl { public boolean isSchemaCaseSensitive() { return false; } + + @Override + public RelDataType deriveSumType(RelDataTypeFactory typeFactory, + RelDataType argumentType) { + // When adding two decimals, Impala sets the precision to 38 + RelDataType rdt = null; + switch (argumentType.getSqlTypeName()) { + case DECIMAL: + if (argumentType.getPrecision() == ScalarType.MAX_PRECISION) { + return argumentType; + } + rdt = typeFactory.createSqlType( + SqlTypeName.DECIMAL, + ScalarType.MAX_PRECISION, + argumentType.getScale()); + break; + case FLOAT: + case DOUBLE: + rdt = typeFactory.createSqlType(SqlTypeName.DOUBLE); + break; + default: + rdt = typeFactory.createSqlType(SqlTypeName.BIGINT); + break; + } + return typeFactory.createTypeWithNullability(rdt, true); + } + + public static RelDataType deriveArithmeticType(RelDataTypeFactory typeFactory, + RelDataType type1, RelDataType type2, ArithmeticExpr.Operator op) { + try { + Type t1 = ImpalaTypeConverter.createImpalaType(type1); + Type t2 = ImpalaTypeConverter.createImpalaType(type2); + // Call out to Impala code to get the correct derived precision on + // arithmetic operations. + if (t1.equals(Type.TIMESTAMP) || t2.equals(Type.TIMESTAMP)) { + return ImpalaTypeConverter.getRelDataType(Type.TIMESTAMP); + } + if (SqlTypeName.INTERVAL_TYPES.contains(type1.getSqlTypeName())) { + return type1; + } + if (SqlTypeName.INTERVAL_TYPES.contains(type2.getSqlTypeName())) { + return type2; + } + + Type retType = TypesUtil.getArithmeticResultType(t1, t2, op, true); + SqlTypeName sqlTypeName = + ImpalaTypeConverter.getRelDataType(retType).getSqlTypeName(); + RelDataType preNullableType = + (sqlTypeName == SqlTypeName.DECIMAL) + ? typeFactory.createSqlType(sqlTypeName, + retType.getPrecision(), retType.getDecimalDigits()) + : typeFactory.createSqlType(sqlTypeName); + boolean nullable = type1.isNullable() && type2.isNullable(); + return typeFactory.createTypeWithNullability(preNullableType, nullable); + } catch (Exception e) { + throw new RuntimeException(e); + } + } + + @Override + public RelDataType deriveDecimalPlusType(RelDataTypeFactory typeFactory, + RelDataType type1, RelDataType type2) { + return deriveArithmeticType(typeFactory, type1, type2, ArithmeticExpr.Operator.ADD); + } + + @Override + public RelDataType deriveDecimalMultiplyType(RelDataTypeFactory typeFactory, + RelDataType type1, RelDataType type2) { + return deriveArithmeticType(typeFactory, type1, type2, + ArithmeticExpr.Operator.MULTIPLY); + } + + @Override + public RelDataType deriveDecimalDivideType(RelDataTypeFactory typeFactory, + RelDataType type1, RelDataType type2) { + return deriveArithmeticType(typeFactory, type1, type2, + ArithmeticExpr.Operator.DIVIDE); + } + + @Override + public RelDataType deriveDecimalModType(RelDataTypeFactory typeFactory, + RelDataType type1, RelDataType type2) { + return deriveArithmeticType(typeFactory, type1, type2, ArithmeticExpr.Operator.MOD); + } + + @Override + public RelDataType deriveAvgAggType(RelDataTypeFactory typeFactory, + RelDataType argumentType) { + try { + switch (argumentType.getSqlTypeName()) { + case DECIMAL: { + // This code is similar to code in Impala's FunctionCallExpr + ScalarType t1 = (ScalarType) ImpalaTypeConverter.createImpalaType(argumentType); + int digitsBefore = t1.decimalPrecision() - t1.decimalScale(); + int digitsAfter = t1.decimalScale(); + int resultScale = Math.max(ScalarType.MIN_ADJUSTED_SCALE, digitsAfter); + int resultPrecision = digitsBefore + resultScale; + return typeFactory.createSqlType(SqlTypeName.DECIMAL, resultPrecision, + resultScale); + } + case TIMESTAMP: { + return argumentType; + } + default: { + return typeFactory.createSqlType(SqlTypeName.DOUBLE); + } + } + } catch (Exception e) { + throw new RuntimeException(e); + } + } } diff --git a/java/calcite-planner/src/test/java/org/apache/impala/CompatibilityTest.java b/java/calcite-planner/src/test/java/org/apache/impala/CompatibilityTest.java new file mode 100755 index 000000000..93b2d881d --- /dev/null +++ b/java/calcite-planner/src/test/java/org/apache/impala/CompatibilityTest.java @@ -0,0 +1,96 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.impala.calcite.service; + +import static org.junit.Assert.assertTrue; + +import org.apache.calcite.rel.type.RelDataType; +import org.apache.commons.lang3.tuple.Pair; +import org.apache.impala.calcite.functions.ImplicitTypeChecker; +import org.apache.impala.calcite.type.ImpalaTypeConverter; +import org.apache.impala.catalog.PrimitiveType; +import org.apache.impala.catalog.ScalarType; +import org.apache.impala.catalog.Type; +import org.apache.impala.catalog.TypeCompatibility; + +import org.junit.Test; + +public class CompatibilityTest { + @Test + public void testCompatibility() { + for (PrimitiveType fromPType : PrimitiveType.values()) { + for (PrimitiveType toPType : PrimitiveType.values()) { + // skip the invalid type since that will never map to/from anything + if (fromPType == PrimitiveType.INVALID_TYPE || + toPType == PrimitiveType.INVALID_TYPE) { + continue; + } + //TODO: support datetime + if (fromPType == PrimitiveType.DATETIME|| + toPType == PrimitiveType.DATETIME) { + continue; + } + // intermediate type not used for Calcite mappings + if (fromPType == PrimitiveType.FIXED_UDA_INTERMEDIATE || + toPType == PrimitiveType.FIXED_UDA_INTERMEDIATE) { + continue; + } + if (fromPType == PrimitiveType.FLOAT && toPType == PrimitiveType.DECIMAL) { + continue; + } + if (fromPType == PrimitiveType.DOUBLE && toPType == PrimitiveType.DECIMAL) { + continue; + } + ScalarType fromType = Type.getDefaultScalarType(fromPType); + ScalarType toType = Type.getDefaultScalarType(toPType); + RelDataType fromRelDataType = ImpalaTypeConverter.getRelDataType(fromType); + RelDataType toRelDataType = ImpalaTypeConverter.getRelDataType(toType); + if (fromRelDataType == null) { + throw new RuntimeException("Couldn't translate " + fromType); + } + if (toRelDataType == null) { + throw new RuntimeException("Couldn't translate " + toType); + } + boolean isSupportedCalcite = ImplicitTypeChecker.supportsImplicitCasting( + fromRelDataType, toRelDataType); + + boolean isSupportedImpala = + Type.isImplicitlyCastable(fromType, toType, TypeCompatibility.DEFAULT); + + // special case float and double to decimal. Calcite allows this because of + // arithmetic operations. A float plus a decimal returns a decimal type. + if ((fromPType == PrimitiveType.FLOAT || fromPType == PrimitiveType.DOUBLE) && + toPType == PrimitiveType.DECIMAL) { + assertTrue("Support mismatch for " + fromPType + " to " + toPType + ". " + + "Impala does" + (isSupportedImpala ? "" : " not") + + " support this conversion." + + "SqlTypeName from is " + fromRelDataType.getSqlTypeName() + + ", SqlTypeName to is " + toRelDataType.getSqlTypeName() + ".", + !isSupportedImpala && isSupportedCalcite); + } else { + assertTrue("Support mismatch for " + fromPType + " to " + toPType + ". " + + "Impala does" + (isSupportedImpala ? "" : " not") + + " support this conversion." + + "SqlTypeName from is " + fromRelDataType.getSqlTypeName() + + ", SqlTypeName to is " + toRelDataType.getSqlTypeName() + ".", + isSupportedImpala == isSupportedCalcite); + } + } + } + } +} diff --git a/testdata/workloads/functional-query/queries/QueryTest/calcite.test b/testdata/workloads/functional-query/queries/QueryTest/calcite.test index f55856bc7..5c29f0639 100644 --- a/testdata/workloads/functional-query/queries/QueryTest/calcite.test +++ b/testdata/workloads/functional-query/queries/QueryTest/calcite.test @@ -174,10 +174,11 @@ select abs(cast(-8 as bigint)); bigint ==== ---- QUERY -# TODO: Char types are not yet supported, see comment in ImpalaValuesRel for details. select 'hello' ----- CATCH -Char type values are not yet supported. +---- RESULTS +'hello' +---- TYPES +string ==== ---- QUERY # Union test @@ -186,10 +187,7 @@ select 3 union select 4; 3 4 ---- TYPES -# IMPALA-13022: Calcite always casts literal numbers as INT. To avoid backward compatibility -# issues, the return type should be TINYINT here. There are many tests in the Impala -# test suite which will have this problem. -int +tinyint ==== ---- QUERY select * from (values (1)) union (values (2), (3)); @@ -198,10 +196,7 @@ select * from (values (1)) union (values (2), (3)); 2 3 ---- TYPES -# IMPALA-13022: Calcite always casts literal numbers as INT. To avoid backward compatibility -# issues, the return type should be TINYINT here. There are many tests in the Impala -# test suite which will have this problem. -int +tinyint ==== ---- QUERY # sort test @@ -487,3 +482,289 @@ order by a.id ---- TYPES int, bigint, bigint ==== +---- QUERY +select bigint_col + bigint_col, int_col + int_col, smallint_col + smallint_col, +tinyint_col + tinyint_col, smallint_col + tinyint_col +from functional.alltypestiny; +---- RESULTS +0,0,0,0,0 +0,0,0,0,0 +0,0,0,0,0 +0,0,0,0,0 +20,2,2,2,2 +20,2,2,2,2 +20,2,2,2,2 +20,2,2,2,2 +---- TYPES +bigint, bigint, int, smallint, int +==== +---- QUERY +select float_col + int_col, float_col + 3.0, 3.0 + 3.0 +from functional.alltypestiny; +---- RESULTS +0.0,3.00000000,6.0 +2.10000002384,4.10000002,6.0 +0.0,3.00000000,6.0 +2.10000002384,4.10000002,6.0 +0.0,3.00000000,6.0 +2.10000002384,4.10000002,6.0 +0.0,3.00000000,6.0 +2.10000002384,4.10000002,6.0 +---- TYPES +double, decimal, decimal +==== +---- QUERY +select bigint_col - bigint_col, int_col - int_col, smallint_col - smallint_col, +tinyint_col - tinyint_col, smallint_col - tinyint_col +from functional.alltypestiny; +---- RESULTS +0,0,0,0,0 +0,0,0,0,0 +0,0,0,0,0 +0,0,0,0,0 +0,0,0,0,0 +0,0,0,0,0 +0,0,0,0,0 +0,0,0,0,0 +---- TYPES +bigint, bigint, int, smallint, int +==== +---- QUERY +select float_col - int_col, float_col - 3.0, 3.0 - 1.8 +from functional.alltypestiny; +---- RESULTS +0.0,-3.00000000,1.2 +0.1000000238418579,-1.89999998,1.2 +0.0,-3.00000000,1.2 +0.1000000238418579,-1.89999998,1.2 +0.0,-3.00000000,1.2 +0.1000000238418579,-1.89999998,1.2 +0.0,-3.00000000,1.2 +0.1000000238418579,-1.89999998,1.2 +---- TYPES +double, decimal, decimal +==== +---- QUERY +select bigint_col * bigint_col, int_col * int_col, smallint_col * smallint_col, +tinyint_col * tinyint_col, smallint_col * tinyint_col +from functional.alltypestiny; +---- RESULTS +0,0,0,0,0 +100,1,1,1,1 +0,0,0,0,0 +100,1,1,1,1 +0,0,0,0,0 +100,1,1,1,1 +0,0,0,0,0 +100,1,1,1,1 +---- TYPES +bigint, bigint, int, smallint, int +==== +---- QUERY +select float_col * int_col, float_col * 3.0, 3.0 * 2.0 +from functional.alltypestiny; +---- RESULTS +0,0,6.00 +1.100000023841858,3.300000071525574,6.00 +0,0,6.00 +1.100000023841858,3.300000071525574,6.00 +0,0,6.00 +1.100000023841858,3.300000071525574,6.00 +0,0,6.00 +1.100000023841858,3.300000071525574,6.00 +---- TYPES +double, double, decimal +==== +---- QUERY +select bigint_col / bigint_col, int_col / int_col, smallint_col / smallint_col, +tinyint_col / tinyint_col +from functional.alltypestiny; +---- RESULTS +1,1,1,1 +NaN,NaN,NaN,NaN +1,1,1,1 +NaN,NaN,NaN,NaN +1,1,1,1 +NaN,NaN,NaN,NaN +1,1,1,1 +NaN,NaN,NaN,NaN +---- TYPES +double, double, double, double +==== +---- QUERY +select float_col / int_col, float_col / 3.0, 3.0 / 2.0 +from functional.alltypestiny; +---- RESULTS +1.100000023841858,0.36666667,1.500000 +NaN,0.00000000,1.500000 +1.100000023841858,0.36666667,1.500000 +NaN,0.00000000,1.500000 +1.100000023841858,0.36666667,1.500000 +NaN,0.00000000,1.500000 +1.100000023841858,0.36666667,1.500000 +NaN,0.00000000,1.500000 +---- TYPES +double, decimal, decimal +==== +---- QUERY +select bigint_col % bigint_col, int_col % int_col, smallint_col % smallint_col, +tinyint_col % tinyint_col, smallint_col % tinyint_col +from functional.alltypestiny; +---- RESULTS +0,0,0,0,0 +NULL,NULL,NULL,NULL,NULL +0,0,0,0,0 +NULL,NULL,NULL,NULL,NULL +0,0,0,0,0 +NULL,NULL,NULL,NULL,NULL +0,0,0,0,0 +NULL,NULL,NULL,NULL,NULL +---- TYPES +bigint, int, smallint, tinyint, smallint +==== +---- QUERY +select float_col % 3.0, 3.0 % 2.0 +from functional.alltypestiny; +---- RESULTS +0.000000000,1.0 +1.100000024,1.0 +0.000000000,1.0 +1.100000024,1.0 +0.000000000,1.0 +1.100000024,1.0 +0.000000000,1.0 +1.100000024,1.0 +---- TYPES +decimal, decimal +==== +---- QUERY +# Union test +select 3 union select 458; +---- RESULTS +3 +458 +---- TYPES +smallint +==== +---- QUERY +# Union test +select 3 union select 458; +---- RESULTS +3 +458 +---- TYPES +smallint +==== +---- QUERY +# case test +select tinyint_col, case tinyint_col when 1 then 5 else 458 end from functional.alltypestiny; +---- RESULTS +0,458 +0,458 +0,458 +0,458 +1,5 +1,5 +1,5 +1,5 +---- TYPES +tinyint,smallint +==== +---- QUERY +# case test +select tinyint_col, case tinyint_col when 1 then 5 when 2 then 7 else 458 end from functional.alltypestiny; +---- RESULTS +0,458 +0,458 +0,458 +0,458 +1,5 +1,5 +1,5 +1,5 +---- TYPES +tinyint,smallint +==== +---- QUERY +# case test +select tinyint_col, case tinyint_col when 0 then 458 else 5 end from functional.alltypestiny; +---- RESULTS +0,458 +0,458 +0,458 +0,458 +1,5 +1,5 +1,5 +1,5 +---- TYPES +tinyint,smallint +==== +---- QUERY +# case test +select tinyint_col, case tinyint_col when 0 then 458 end from functional.alltypestiny; +---- RESULTS +0,458 +0,458 +0,458 +0,458 +1,NULL +1,NULL +1,NULL +1,NULL +---- TYPES +tinyint,smallint +==== +---- QUERY +# case test other format (the calcite rexnode should be the same) +select tinyint_col, case when tinyint_col=00 then 458 else 5 end from functional.alltypestiny; +---- RESULTS +0,458 +0,458 +0,458 +0,458 +1,5 +1,5 +1,5 +1,5 +---- TYPES +tinyint,smallint +==== +---- QUERY +# or test +select (tinyint_col = 459) or (tinyint_col = 458) or (tinyint_col = 1) from functional.alltypestiny; +---- RESULTS +false +false +false +false +true +true +true +true +---- TYPES +boolean +==== +---- QUERY +# and test +select (tinyint_col != 459) and (tinyint_col != 458) and (tinyint_col = 1) from functional.alltypestiny; +---- RESULTS +false +false +false +false +true +true +true +true +---- TYPES +boolean +==== +---- QUERY +# sum cast tinyint agg test +select sum(tinyint_col) from functional.alltypestiny; +---- RESULTS +4 +---- TYPES +bigint +====
