[
https://issues.apache.org/jira/browse/SPARK-55569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125309#comment-18125309
]
Shrirang Mhalgi edited comment on SPARK-55569 at 10/9/26 1:09 AM:
------------------------------------------------------------------
Renamed the ticket to match the PR title per reviewer feedback. The original
ask was for a filter pushed down to the source, but a logical filter cannot be
pushed below GROUP BY without removing group-by keys whose only rows have
non-matching pivot values. The PR instead collapses non-matching pivot column
values to null in the group-by key, reducing the number of firstAgg groups from
O(distinct values) to O(1) per group-by key while preserving all group-by keys
in the output.
was (Author: JIRAUSER313104):
A logical filter cannot be pushed below GROUP BY without removing group-by keys
whose only rows have non-matching pivot values. Instead, this optimization
collapses non-matching pivot column values to null in the group-by key,
reducing the number of firstAgg groups from O(distinct values) to O(1) per
group-by key while preserving all group-by keys in the output.
> Collapse non-matching pivot column values in fast-path firstAgg
> ---------------------------------------------------------------
>
> Key: SPARK-55569
> URL: https://issues.apache.org/jira/browse/SPARK-55569
> Project: Spark
> Issue Type: Wish
> Components: Optimizer
> Affects Versions: 4.1.1
> Reporter: Malthe Borch
> Priority: Major
> Labels: pull-request-available
>
> When pivoting on a set of values, this is potentially a subset of all values.
> Nonetheless, there's no predicate pushdown to the source.
> {code:sql}
> SELECT * FROM person
> PIVOT (
> SUM(age) AS a, AVG(class) AS c
> FOR name IN ('John' AS john, 'Mike' AS mike)
> );
> {code}
> In this example there is no predicate pushdown to filter the {{person}} table
> on the specified names.
> The workaround is to repeat the same set of values in a WHERE-clause before
> the pivot clause, but this is redundant. In addition, I think it can be
> argued that you expect the optimizer to do this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
