snmvaughan opened a new pull request, #7265: URL: https://github.com/apache/datafusion-comet/pull/7265
## Which issue does this PR close? Backport of #6989 (squash commit 3565f2f9b) to `branch-1.1`, for #6768. ## Rationale for this change `branch-1.1` has the bug #6989 fixes on `main`. Spark 4.0.5, 4.1.4 and 4.2.1 normalize each flat `FLOAT` or `DOUBLE` element before `arrays_overlap` looks it up (SPARK-59602), so `-0.0` and `0.0` are one value, and Spark 4.2.0 gets the same answer by normalizing the arguments in the plan (SPARK-54918). On `branch-1.1`, `CometArraysOverlap` treats floating-point elements as compatible and runs the native `spark_arrays_overlap`, which keeps the zeros apart. So on those releases `arrays_overlap(array(0.0), array(-0.0))` returns `false` where Spark returns `true`, by default. See #6768. ## What changes are included in this PR? `git cherry-pick -x` of 3565f2f9b, adapted because `branch-1.1` predates two changes #6989 builds on: - `branch-1.1` has no `ArraySetSupport.normalizesFloats` (#6563), so this adds it next to `normalizesArgumentsInPlan`, for `arrays_overlap` only. `array_distinct` and `array_union` keep `branch-1.1`'s 4.2.0-only check. - `arrays_overlap.rs` takes `normalize_float` from `math_funcs::internal` and keeps `branch-1.1`'s own NaN keying, since `branch-1.1` predates the `float_semantics` module (#6400). Otherwise the native change is the same: `SparkArraysOverlap` has a `normalize_floats` flag and is registered as both `spark_arrays_overlap` and `spark_arrays_overlap_normalized`, and `CometArraysOverlap.nativeFunction` picks the variant by the Spark version. - The new test pins the version boundaries itself, since `branch-1.1` doesn't have the test that does on `main`. - The user-guide floating-point page gets the `arrays_overlap` section only. The contributor-guide pages are dropped, since that guide isn't published from release branches. Backports to other branches: none for `branch-1.2`, which already has #6989. ## How are these changes tested? The native tests come with the cherry-pick, and the flat `FLOAT` and `DOUBLE` tests check both variants. On this branch, all 978 `datafusion-comet-spark-expr` unit tests pass, `cargo fmt --check`, `cargo clippy --all-targets --workspace -- -D warnings` and spotless are clean, and the Scala code compiles. I haven't run the Scala suites locally. `arrays_overlap native variant by Spark version` covers the version boundaries and the variant `convert` asks for. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
