geoffreyclaude opened a new pull request, #25187: URL: https://github.com/apache/datafusion/pull/25187
## Which issue does this PR close? - Part of #19241. - Stacked on [#25186](https://github.com/apache/datafusion/pull/25186), which is stacked on [#24088](https://github.com/apache/datafusion/pull/24088). ## Rationale for this change The expression simplifier currently rewrites singleton `IN` lists, and two- or three-item lists on a column, into left-deep `OR` / `AND` comparison chains. That rule predates the specialized static filters in this series. For supported types, rewriting two- and three-item lists now prevents them from reaching a substantially faster branchless physical path. The rewrite should therefore depend on the available physical representation, not only on list length. This PR keeps singleton-to-equality simplification, retains static two- and three-item `InList` expressions when a specialized filter can evaluate them, and preserves the existing comparison rewrite for dynamic or generic lists. Although the singleton `InList` path is faster for several representations in isolated evaluation, equality remains the canonical logical form and can benefit other optimizer and pushdown paths that this microbenchmark does not measure. ## What changes are included in this PR? - Gives the short-list simplifier access to type information. - Retains two- and three-item constant lists for supported primitive, temporal, decimal, fixed-size-binary, and inline byte-view representations, including dictionary wrappers. - Keeps the existing `OR` / `AND` expansion for generic types, out-of-line byte views, and non-constant lists. - Decouples the preimage rewrite limit from the short-list physical-filter decision; its existing three-value limit is unchanged. - Adds a SQL-planned benchmark for list sizes one through four and updates affected plan snapshots. ## Are these changes tested? Coverage includes the retained and expanded boundaries; `IN` and `NOT IN`; typed nulls; dynamic lists; nested dictionaries; supported and unsupported fixed-size-binary widths; inline and out-of-line `Utf8View` / `BinaryView` values; and the independent preimage limit. Focused validation: - `cargo test -p datafusion-optimizer --lib` (796 tests) - All 11 affected SQL logic test files - `cargo test -p datafusion-substrait roundtrip_inlist` (5 tests) - `cargo test -p datafusion-sqllogictest --test sqllogictests -- negative_zero` - `cargo check -p datafusion --bench in_list_rewrite` - `cargo bench -p datafusion --bench in_list_rewrite -- --noplot` ## Are there any user-facing changes? No query results or public APIs change. Optimized plans retain two- and three-item `InList` expressions for supported constant-list representations instead of expanding them into comparison chains. ## Local benchmark snapshot The new benchmark parses and analyzes both forms from SQL, asserts their physical shapes and equal outputs, then measures only physical-expression evaluation. It covers five representative types; list sizes 1-4; batches of 1, 64, and 8,192 rows; input and list nulls; miss, balanced, and skewed match rates; and both `IN` and `NOT IN`: 300 paired scenarios and 600 Criterion functions in total. The table focuses on the two sizes whose optimizer policy changes. Each ratio is the median `comparison chain / retained InList` time across 16 scenarios; values above 1 favor retaining `InList`. | Representation | 2 items | Wins | 3 items | Wins | Decision | |---|---:|---:|---:|---:|---| | `FixedSizeBinary(8)` | 17.26x | 16/16 | 17.99x | 16/16 | Retain | | `Float64` | 3.29x | 16/16 | 3.96x | 16/16 | Retain | | `Int32` | 1.85x | 16/16 | 2.27x | 16/16 | Retain | | inline `Utf8View` | 1.34x | 14/16 | 1.35x | 14/16 | Retain | | generic `Utf8` | 1.25x | 11/16 | 1.77x | 12/16 | Keep `OR` / `AND` | Generic `Utf8` remains mixed, including regressions for skewed and null-containing lists, so this PR deliberately leaves it on the existing comparison rewrite. List sizes one and four serve as boundary controls; their optimizer policy is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
