[
https://issues.apache.org/jira/browse/CALCITE-7687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Etienne Pelissier updated CALCITE-7687:
---------------------------------------
Description:
{{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
{code:java}
public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
@Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
rel.getGroupSet(), // [2]
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPred =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// [1]
Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
...
}
{code}
{{[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
predicate)}} method:
{code:java}
public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
mq,
ImmutableBitSet groupKey, @Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
ImmutableBitSet.range(rel.getGroupCount()),
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPreds =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// set the bits as they correspond to the child input
ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
// [1]
Double distinctRowCount =
mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
...
}
{code}
both currently pass down the predicate to their input [1] without considering
any possible translation, since an {{Aggregate}} derives its row type as
{{(group keys..., aggregate calls...)}} and so its output field {{i}} is input
field {{groupSet.nth(i\)}}; hence when the Aggregate's input analyzes the
predicate, it can end up reading a different column from the one the predicate
names.
The consequence is silent rather than an exception, like the tests attached to
the first two comments, where, in the first comment, after
{{RelMdSelectivity#getSelectivity(Aggregate)}} we reach
{{RelMdSelectivity#getSelectivity(TableScan)}} and this method hands the
predicate to the {{BuiltInMetadata.Selectivity.Handler}} that the table exposes
through {{RelOptTable#unwrap}}, which reports the null fraction of a field ($1)
that is not the one the predicate names. This $1 is the Aggregate's second
group key, which is field ($2) of the scan underneath it. The second comment
shows the same thing one method over, where the
{{BuiltInMetadata.DistinctRowCount.Handler}} receives that unconverted ($1)
next to a {{groupKey}} that {{RelMdUtil.setAggChildKeys}} did convert.
Note I: [CALCITE-4414|https://issues.apache.org/jira/browse/CALCITE-4414] is
very closely related to this ticket.
Note II: in a similar situation, {{RelMdSelectivity#getSelectivity(Project)}}
uses {{RelOptUtil.pushPastProject}} and
{{RelMdSelectivity#getSelectivity(Calc)}} uses {{RelOptUtil.pushPastCalc}}
(which "Converts an expression that is based on the output fields of a Calc to
an equivalent expression on the Calc's input fields.") to convert the predicate
before passing it to the input. There is no equivalent
{{RelOptUtil.pushPastAggregate}}, even though {{FilterAggregateTransposeRule}}
already computes that same mapping inline in order to push a {{Filter}} past an
{{Aggregate}}.
was:
{{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
{code:java}
public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
@Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
rel.getGroupSet(), // [2]
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPred =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// [1]
Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
...
}
{code}
{{[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
predicate)}} method:
{code:java}
public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
mq,
ImmutableBitSet groupKey, @Nullable RexNode predicate) {
...
RelOptUtil.splitFilters(
ImmutableBitSet.range(rel.getGroupCount()),
predicate,
pushable,
notPushable);
final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
RexNode childPreds =
RexUtil.composeConjunction(rexBuilder, pushable, true);
// set the bits as they correspond to the child input
ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
// [1]
Double distinctRowCount =
mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
...
}
{code}
both currently pass down the predicate to their input [1] without considering
any possible translation, since an {{Aggregate}} derives its row type as
{{(group keys..., aggregate calls...)}} and so its output field {{i}} is input
field {{groupSet.nth(i\)}}; hence when the Aggregate's input analyzes the
predicate, it can end up reading a different column from the one the predicate
names.
The consequence is silent rather than an exception, like the tests attached to
the first two comments, where after
{{RelMdSelectivity#getSelectivity(Aggregate)}} we reach
{{RelMdSelectivity#getSelectivity(TableScan)}} and this method hands the
predicate to the {{BuiltInMetadata.Selectivity.Handler}} that the table exposes
through {{RelOptTable#unwrap}}, which reports the null fraction of a field ($1)
that is not the one the predicate names. This $1 is the Aggregate's second
group key, which is field ($2) of the scan underneath it.
Note I: [CALCITE-4414|https://issues.apache.org/jira/browse/CALCITE-4414] is
very closely related to this ticket.
Note II: in a similar situation, {{RelMdSelectivity#getSelectivity(Project)}}
uses {{RelOptUtil.pushPastProject}} and
{{RelMdSelectivity#getSelectivity(Calc)}} uses {{RelOptUtil.pushPastCalc}}
(which "Converts an expression that is based on the output fields of a Calc to
an equivalent expression on the Calc's input fields.") to convert the predicate
before passing it to the input. There is no equivalent
{{RelOptUtil.pushPastAggregate}}, even though {{FilterAggregateTransposeRule}}
already computes that same mapping inline in order to push a {{Filter}} past an
{{Aggregate}}.
> RelMdSelectivity and RelMdDistinctRowCount for Aggregate can propagate a
> predicate with wrong references
> --------------------------------------------------------------------------------------------------------
>
> Key: CALCITE-7687
> URL: https://issues.apache.org/jira/browse/CALCITE-7687
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.42.0
> Reporter: Etienne Pelissier
> Assignee: Etienne Pelissier
> Priority: Minor
> Labels: in-progress, pull-request-available
>
> {{[RelMdSelectivity#getSelectivity|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdSelectivity.java#L178](Aggregate
> rel, RelMetadataQuery mq, @Nullable RexNode predicate)}} method:
> {code:java}
> public @Nullable Double getSelectivity(Aggregate rel, RelMetadataQuery mq,
> @Nullable RexNode predicate) {
> ...
> RelOptUtil.splitFilters(
> rel.getGroupSet(), // [2]
> predicate,
> pushable,
> notPushable);
> final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
> RexNode childPred =
> RexUtil.composeConjunction(rexBuilder, pushable, true);
> // [1]
> Double selectivity = mq.getSelectivity(rel.getInput(), childPred);
> ...
> }
> {code}
> {{[RelMdDistinctRowCount#getDistinctRowCount|https://github.com/apache/calcite/blob/0211999427e294114db8f6b58cad95870cc13ccb/core/src/main/java/org/apache/calcite/rel/metadata/RelMdDistinctRowCount.java#L168](Aggregate
> rel, RelMetadataQuery mq, ImmutableBitSet groupKey, @Nullable RexNode
> predicate)}} method:
> {code:java}
> public @Nullable Double getDistinctRowCount(Aggregate rel, RelMetadataQuery
> mq,
> ImmutableBitSet groupKey, @Nullable RexNode predicate) {
> ...
> RelOptUtil.splitFilters(
> ImmutableBitSet.range(rel.getGroupCount()),
> predicate,
> pushable,
> notPushable);
> final RexBuilder rexBuilder = rel.getCluster().getRexBuilder();
> RexNode childPreds =
> RexUtil.composeConjunction(rexBuilder, pushable, true);
> // set the bits as they correspond to the child input
> ImmutableBitSet.Builder childKey = ImmutableBitSet.builder();
> RelMdUtil.setAggChildKeys(groupKey, rel, childKey);
> // [1]
> Double distinctRowCount =
> mq.getDistinctRowCount(rel.getInput(), childKey.build(), childPreds);
> ...
> }
> {code}
> both currently pass down the predicate to their input [1] without considering
> any possible translation, since an {{Aggregate}} derives its row type as
> {{(group keys..., aggregate calls...)}} and so its output field {{i}} is
> input field {{groupSet.nth(i\)}}; hence when the Aggregate's input analyzes
> the predicate, it can end up reading a different column from the one the
> predicate names.
> The consequence is silent rather than an exception, like the tests attached
> to the first two comments, where, in the first comment, after
> {{RelMdSelectivity#getSelectivity(Aggregate)}} we reach
> {{RelMdSelectivity#getSelectivity(TableScan)}} and this method hands the
> predicate to the {{BuiltInMetadata.Selectivity.Handler}} that the table
> exposes through {{RelOptTable#unwrap}}, which reports the null fraction of a
> field ($1) that is not the one the predicate names. This $1 is the
> Aggregate's second group key, which is field ($2) of the scan underneath it.
> The second comment shows the same thing one method over, where the
> {{BuiltInMetadata.DistinctRowCount.Handler}} receives that unconverted ($1)
> next to a {{groupKey}} that {{RelMdUtil.setAggChildKeys}} did convert.
> Note I: [CALCITE-4414|https://issues.apache.org/jira/browse/CALCITE-4414] is
> very closely related to this ticket.
> Note II: in a similar situation, {{RelMdSelectivity#getSelectivity(Project)}}
> uses {{RelOptUtil.pushPastProject}} and
> {{RelMdSelectivity#getSelectivity(Calc)}} uses {{RelOptUtil.pushPastCalc}}
> (which "Converts an expression that is based on the output fields of a Calc
> to an equivalent expression on the Calc's input fields.") to convert the
> predicate before passing it to the input. There is no equivalent
> {{RelOptUtil.pushPastAggregate}}, even though
> {{FilterAggregateTransposeRule}} already computes that same mapping inline in
> order to push a {{Filter}} past an {{Aggregate}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
