[
https://issues.apache.org/jira/browse/SPARK-55569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shrirang Mhalgi updated SPARK-55569:
------------------------------------
Summary: Collapse non-matching pivot column values in fast-path firstAgg
(was: Filter pushdown in pivot aggregation)
> Collapse non-matching pivot column values in fast-path firstAgg
> ---------------------------------------------------------------
>
> Key: SPARK-55569
> URL: https://issues.apache.org/jira/browse/SPARK-55569
> Project: Spark
> Issue Type: Wish
> Components: Optimizer
> Affects Versions: 4.1.1
> Reporter: Malthe Borch
> Priority: Major
> Labels: pull-request-available
>
> When pivoting on a set of values, this is potentially a subset of all values.
> Nonetheless, there's no predicate pushdown to the source.
> {code:sql}
> SELECT * FROM person
> PIVOT (
> SUM(age) AS a, AVG(class) AS c
> FOR name IN ('John' AS john, 'Mike' AS mike)
> );
> {code}
> In this example there is no predicate pushdown to filter the {{person}} table
> on the specified names.
> The workaround is to repeat the same set of values in a WHERE-clause before
> the pivot clause, but this is redundant. In addition, I think it can be
> argued that you expect the optimizer to do this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
