[
https://issues.apache.org/jira/browse/CALCITE-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Hu updated CALCITE-7101:
---------------------------
Description:
We are trying to upgrade Apache Calcite version in our project (Apache Beam
[https://github.com/apache/beam/pull/35588]). Found a potential breaking change.
Prior to Calcite <=1.33, the following query works for the test
{quote}"select * from CUSTOMER "
+ " where exists ( "
+ " select * from ORDERS "
+ " where o_custkey = c_custkey )";
{quote}
However, since Calcite 1.34 (to 1.38), the following error is observed
{quote}java.lang.RuntimeException: Error while applying rule FilterToCalcRule,
args [rel#55:LogicalFilter.NONE(input=RelSubset#54,condition=EXISTS(
Unknown macro: \{ LogicalFilter(condition=[=($1, $cor0.c_custkey)])
BeamIOSourceRel(table=[[beam, ORDERS]]) }
),variablesSet=[$cor0])]
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)
at
org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:523)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:317)
at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)
at
org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)
Caused by: java.lang.ClassCastException: class
org.apache.calcite.rex.RexSubQuery cannot be cast to class
org.apache.calcite.rex.RexLocalRef (org.apache.calcite.rex.RexSubQuery and
org.apache.calcite.rex.RexLocalRef are in unnamed module of loader 'app')
at
org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:304)
{quote}
It is appeared due to CALCITE-6874, which is fixed in Calcite 1.39. However the
fix does not fix our use case In 1.39/1.40, a different error is seen:
{quote}org.apache.beam.sdk.extensions.sql.impl.SqlConversionException: Unable
to convert query select * from CUSTOMER where exists ( select * from ORDERS
where o_custkey = c_custkey )
at
app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:214)
at
app//org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:116)
Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are
not enough rules to produce a node with desired properties:
convention=BEAM_LOGICAL.
Missing conversion is LogicalFilter[convention: NONE -> BEAM_LOGICAL]
at
app//org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit(RelSubset.java:718)
at
app//org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(RelSubset.java:391)
at
app//org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:535)
at app//org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:353)
at app//org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)
at
app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)
... 48 more
{quote}
Taking a closer look, In Calcite<=1.33 the query is parsed (planner.parse)
as
{quote}LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])
LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2],
o_custkey=[CAST($3):INTEGER], $f1=[CAST($4):BOOLEAN])
LogicalJoin(condition=[=($0, $3)], joinType=[inner])
BeamIOSourceRel(table=[[beam, CUSTOMER]])
LogicalAggregate(group=[\\{0}], agg#0=[MIN($1)])
LogicalProject(o_custkey=[$1], $f0=[true])
BeamIOSourceRel(table=[[beam, ORDERS]])
{quote}
Then send to planner.rel(). After Calcite 1.34, the query parsed differently
{quote}LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])
LogicalFilter(condition=[EXISTS({ LogicalFilter(condition=[=($1,
$cor0.c_custkey)]) BeamIOSourceRel(table=[[beam, ORDERS]]) })],
variablesSet=[[$cor0]]) BeamIOSourceRel(table=[[beam, CUSTOMER]])
{quote}
such that a LogicalFilter no longer converted to LogicalJoin. Then later on
planner.rel() fails.
What might have caused this change (between Calcite 1.33 -> 1.34). This is
currently blocking our upgrade. Is it possible to mitigate this without
implementing a custom "LogicalFilter[convention: NONE -> BEAM_LOGICAL]" rule
such that Calcite still knows to apply FilterToJoinRule?
was:
We are trying to upgrade Apache Calcite version in our project (Apache Beam
https://github.com/apache/beam/pull/35588). Found a potential breaking change.
Prior to Calcite <=1.33, the following query works for the test
{quote}"select * from CUSTOMER "
+ " where exists ( "
+ " select * from ORDERS "
+ " where o_custkey = c_custkey )";{quote}
However, since Calcite 1.34 (to 1.38), the following error is observed
{quote}java.lang.RuntimeException: Error while applying rule FilterToCalcRule,
args [rel#55:LogicalFilter.NONE(input=RelSubset#54,condition=EXISTS({
LogicalFilter(condition=[=($1, $cor0.c_custkey)])
BeamIOSourceRel(table=[[beam, ORDERS]])
}),variablesSet=[$cor0])]
at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)
at
org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:523)
at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:317)
at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)
at
org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)
Caused by: java.lang.ClassCastException: class
org.apache.calcite.rex.RexSubQuery cannot be cast to class
org.apache.calcite.rex.RexLocalRef (org.apache.calcite.rex.RexSubQuery and
org.apache.calcite.rex.RexLocalRef are in unnamed module of loader 'app')
at
org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:304)
{quote}
It is appeared due to CALCITE-6874, which is fixed in Calcite 1.39. However the
fix does not fix our use case In 1.39/1.40, a different error is seen:
{quote}org.apache.beam.sdk.extensions.sql.impl.SqlConversionException: Unable
to convert query select * from CUSTOMER where exists ( select * from ORDERS
where o_custkey = c_custkey )
at
app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:214)
at
app//org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:116)
Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There are
not enough rules to produce a node with desired properties:
convention=BEAM_LOGICAL.
Missing conversion is LogicalFilter[convention: NONE -> BEAM_LOGICAL]
at
app//org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit(RelSubset.java:718)
at
app//org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(RelSubset.java:391)
at
app//org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:535)
at app//org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:353)
at app//org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)
at
app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)
... 48 more
{quote}
Taking a closer look, In Calcite<=1.33 the query is parsed (planner.parse)
as
{quote}LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])
LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2],
o_custkey=[CAST($3):INTEGER], $f1=[CAST($4):BOOLEAN])
LogicalJoin(condition=[=($0, $3)], joinType=[inner])
BeamIOSourceRel(table=[[beam, CUSTOMER]])
LogicalAggregate(group=[\{0}], agg#0=[MIN($1)])
LogicalProject(o_custkey=[$1], $f0=[true])
BeamIOSourceRel(table=[[beam, ORDERS]])
{quote}
Then send to planner.rel(). After Calcite 1.34, the query parsed differently
{quote}LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])
LogicalFilter(condition=[EXISTS(\{ LogicalFilter(condition=[=($1,
$cor0.c_custkey)]) BeamIOSourceRel(table=[[beam, ORDERS]]) })],
variablesSet=[[$cor0]]) BeamIOSourceRel(table=[[beam, CUSTOMER]]){quote}
such that a LogicalFilter no longer converted to LogicalJoin. Then later on
planner.rel() fails.
What might have caused this change (between Calcite 1.33 -> 1.34). This is
currently blocking upgrade. Is it possible to mitigate this without
implementing a custom "LogicalFilter[convention: NONE -> BEAM_LOGICAL]" rule
such that Calcite still knows to apply FilterToJoinRule?
> Query parsed to LogicalJoin changed to LogicalFilter causing
> ClassCastException then CannotPlanException in newer versions
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: CALCITE-7101
> URL: https://issues.apache.org/jira/browse/CALCITE-7101
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.34.0, 1.39.0
> Environment: Java11
> Reporter: Yi Hu
> Priority: Major
>
> We are trying to upgrade Apache Calcite version in our project (Apache Beam
> [https://github.com/apache/beam/pull/35588]). Found a potential breaking
> change.
>
> Prior to Calcite <=1.33, the following query works for the test
>
> {quote}"select * from CUSTOMER "
> + " where exists ( "
> + " select * from ORDERS "
> + " where o_custkey = c_custkey )";
> {quote}
>
> However, since Calcite 1.34 (to 1.38), the following error is observed
>
> {quote}java.lang.RuntimeException: Error while applying rule
> FilterToCalcRule, args
> [rel#55:LogicalFilter.NONE(input=RelSubset#54,condition=EXISTS(
> Unknown macro: \{ LogicalFilter(condition=[=($1, $cor0.c_custkey)])
> BeamIOSourceRel(table=[[beam, ORDERS]]) }
> ),variablesSet=[$cor0])]
> at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:250)
> at
> org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:59)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:523)
> at org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:317)
> at org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)
> at
> org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)
> Caused by: java.lang.ClassCastException: class
> org.apache.calcite.rex.RexSubQuery cannot be cast to class
> org.apache.calcite.rex.RexLocalRef (org.apache.calcite.rex.RexSubQuery and
> org.apache.calcite.rex.RexLocalRef are in unnamed module of loader 'app')
> at
> org.apache.calcite.rex.RexProgramBuilder.registerInput(RexProgramBuilder.java:304)
> {quote}
>
> It is appeared due to CALCITE-6874, which is fixed in Calcite 1.39. However
> the fix does not fix our use case In 1.39/1.40, a different error is seen:
>
> {quote}org.apache.beam.sdk.extensions.sql.impl.SqlConversionException: Unable
> to convert query select * from CUSTOMER where exists ( select * from ORDERS
> where o_custkey = c_custkey )
> at
> app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:214)
> at
> app//org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:116)
>
> Caused by: org.apache.calcite.plan.RelOptPlanner$CannotPlanException: There
> are not enough rules to produce a node with desired properties:
> convention=BEAM_LOGICAL.
> Missing conversion is LogicalFilter[convention: NONE -> BEAM_LOGICAL]
> at
> app//org.apache.calcite.plan.volcano.RelSubset$CheapestPlanReplacer.visit(RelSubset.java:718)
> at
> app//org.apache.calcite.plan.volcano.RelSubset.buildCheapestPlan(RelSubset.java:391)
> at
> app//org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:535)
> at
> app//org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:353)
> at app//org.apache.calcite.prepare.PlannerImpl.transform(PlannerImpl.java:385)
> at
> app//org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:210)
> ... 48 more
> {quote}
>
> Taking a closer look, In Calcite<=1.33 the query is parsed (planner.parse)
> as
>
> {quote}LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])
> LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2],
> o_custkey=[CAST($3):INTEGER], $f1=[CAST($4):BOOLEAN])
> LogicalJoin(condition=[=($0, $3)], joinType=[inner])
> BeamIOSourceRel(table=[[beam, CUSTOMER]])
> LogicalAggregate(group=[\\{0}], agg#0=[MIN($1)])
> LogicalProject(o_custkey=[$1], $f0=[true])
> BeamIOSourceRel(table=[[beam, ORDERS]])
> {quote}
>
> Then send to planner.rel(). After Calcite 1.34, the query parsed differently
>
> {quote}LogicalProject(c_custkey=[$0], c_acctbal=[$1], c_city=[$2])
> LogicalFilter(condition=[EXISTS({ LogicalFilter(condition=[=($1,
> $cor0.c_custkey)]) BeamIOSourceRel(table=[[beam, ORDERS]]) })],
> variablesSet=[[$cor0]]) BeamIOSourceRel(table=[[beam, CUSTOMER]])
> {quote}
>
> such that a LogicalFilter no longer converted to LogicalJoin. Then later on
> planner.rel() fails.
>
> What might have caused this change (between Calcite 1.33 -> 1.34). This is
> currently blocking our upgrade. Is it possible to mitigate this without
> implementing a custom "LogicalFilter[convention: NONE -> BEAM_LOGICAL]" rule
> such that Calcite still knows to apply FilterToJoinRule?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
