adriangb opened a new issue, #25897:
URL: https://github.com/apache/datafusion/issues/25897
Follow-up to #25838 and #25792.
`PullUpCorrelatedExpr` now pulls a correlated filter above a `Window` only
when the filter reads plain `PARTITION BY` columns. Any other correlated filter
below a window keeps the subquery correlated, and the query fails with
"Physical plan does not support logical expression …". For example (#25792):
```sql
SELECT t1.id, sub.id, sub.rn
FROM t1, LATERAL (
SELECT t2.id, row_number() OVER (ORDER BY t2.id) AS rn
FROM (SELECT * FROM t2 WHERE t2.t1_id = t1.id) AS t2
) AS sub;
```
**Proposal:** when every correlated conjunct below the window has the form
`inner_col = <outer expr>`, it fixes `inner_col` for each outer row. We can
then add `inner_col` to the `PARTITION BY` of every window function and pull
the filter above the window. Spark (`DecorrelateInnerQuery`, `Window` case) and
DuckDB (`FlattenDependentJoins::PushDownWindow`) do this. A Projection on top
restores the original window column names.
**Caveats:**
- Only a plain column equality qualifies. `can_pullup_over_aggregation` also
accepts `CAST(col AS …) = outer`, and a lossy cast would map several `col`
values to one outer row, so partitioning by `col` would give wrong results.
- Skip it when the window's input needs the count-bug handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
