Sunninsky opened a new pull request, #27320: URL: https://github.com/apache/datafusion/pull/27320
## Which issue does this PR close? - Part of #26868 (the 55.3.0 patch release tracking issue). - Closes #25471 on `branch-55`. ## Rationale for this change On 55.2.0, `LEAD(v, N) IGNORE NULLS` with `N >= 2` and negative-offset `LAG` with `IGNORE NULLS` return NULL or the default value when a NULL falls inside the lookahead, although the requested non-null row exists. With window LIMIT pushdown enabled, the input can also be cut before the requested row. These are wrong query results, which makes this eligible for backport as a correctness fix. #25472 fixed it on `main`. It is not in 55.1.0 or 55.2.0 and first ships in 56.0.0. It was requested for 55.3.0 on #26868. ## What changes are included in this PR? - A `git cherry-pick -x` of 84acf498cb1e616bd43b1b9d0e3d2f7bdfa86bce (#25472, by @lyne7-sc). It applied without conflicts and this PR changes nothing else. - `datafusion/functions-window/src/lead_lag.rs`: the stateful evaluation resumes scanning after the last cached non-null row until enough candidates are available or the range ends. - `datafusion/physical-plan/src/windows/mod.rs`: `limit_effect` returns `LimitEffect::Unknown` for an `IGNORE NULLS` window UDF expression, since non-null offsets cannot bound the number of input rows needed. - Regression tests in `window.slt` (NULL gaps and batch boundaries) and `window_limits.slt` (LIMIT pushdown). ## Are these changes tested? Yes, by the tests that #25472 added. On this branch (Rust 1.95.0) I ran: - `cargo test -p datafusion-functions-window`: 18 passed. - `cargo test -p datafusion-sqllogictest --test sqllogictests -- window`: all 7 matching files pass, including `window.slt` and `window_limits.slt` with the new cases. ## Are there any user-facing changes? Affected `IGNORE NULLS` window queries return the requested row instead of NULL or the default, as on `main`. No public API change. A window function with `IGNORE NULLS` no longer lets window LIMIT pushdown (`datafusion.optimizer.enable_window_limits`) cut the input, so the plan of such a query can differ. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
