Copilot commented on code in PR #25865:
URL: https://github.com/apache/datafusion/pull/25865#discussion_r4140371350
##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -1108,21 +1115,27 @@ impl OptimizerRule for PushDownFilter {
let mut keep_predicates = vec![];
let mut push_predicates = vec![];
for expr in predicates {
- let cols = expr.column_refs();
- if cols.iter().all(|c|
potential_partition_keys.contains(c)) {
+ // A volatile predicate has to stay above the window.
Pushing it
+ // changes which rows the window function sees, and so the
value
+ // it computes for the rows that do survive. Checking this
first
+ // also covers a volatile predicate that reads no columns
at all,
+ // such as `random() < 0.5`, which would otherwise satisfy
the
+ // partition-key test vacuously.
+ if !expr.is_volatile()
+ && reads_only_partition_keys(&expr,
&potential_partition_keys)?
+ {
Review Comment:
Keep subquery-containing predicates above this window. `Expr::apply` does
not enter scalar/EXISTS subqueries (or the subquery plans in `IN` and set
comparisons), and `is_volatile()` cannot see functions inside them. For
example, with `PARTITION BY a + b`, `a + b > (SELECT ... WHERE inner.x =
outer.a)` now matches the expression key and can be pushed even if
scalar-subquery decorrelation was skipped; `outer.a` can vary within a
partition, so filtering before the window changes its result. The old
column-key check kept this expression above the window. Reject subqueries here
unless their row dependencies can be checked, and add a regression test for a
subquery that remains in the filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
