geoffreyclaude opened a new pull request, #25186: URL: https://github.com/apache/datafusion/pull/25186
## Which issue does this PR close? - Part of #19241. - Stacked on [#24088](https://github.com/apache/datafusion/pull/24088). ## Rationale for this change DataFusion equality treats `+0.0` and `-0.0` as equal, but the optimized `IN` filters introduced in this series compare or hash floating-point bit patterns. Because the two zeros have different encodings, an `IN` predicate could therefore return a different result depending on whether the optimizer retained it or rewrote it into equality comparisons. This PR makes top-level `Float16`, `Float32`, and `Float64` `IN` predicates follow the same signed-zero equality as ordinary comparisons. Static filters materialize both zero encodings once while building the list, keeping lookup on the row path branch-free. Non-static lists normalize evaluated floating operands once per batch. Distinct NaN payloads remain distinct, preserving the existing bit-equality behavior for NaNs. ## What changes are included in this PR? - Adds both signed-zero encodings to branchless floating filters when either zero occurs in the static list. - Applies the same rule to the `Float16` bitmap and `Float32` / `Float64` hash-set filters. - Extends the shared zero normalizer to dictionary-wrapped floating arrays and scalar values for non-static lists and equality rewrites. - Preserves null behavior, dictionary inputs, `IN`, and `NOT IN`. ## Are these changes tested? The SQL logic tests cover both signed-zero directions for `Float16`, `Float32`, `Float64`, and dictionary-encoded floats; list lengths on both sides of the short-list rewrite threshold; and both `IN` and `NOT IN`. Focused validation: - `cargo test -p datafusion-common normalize_float_zero` - `cargo test -p datafusion-physical-expr in_list` (89 tests) - `cargo test -p datafusion-sqllogictest --test sqllogictests -- negative_zero` ## Are there any user-facing changes? Yes. Floating-point `IN` and `NOT IN` now treat `+0.0` and `-0.0` as equal, consistently with ordinary DataFusion equality. There are no public API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
