andygrove opened a new pull request, #6521: URL: https://github.com/apache/datafusion-comet/pull/6521
## Which issue does this PR close? Part of #6385: the "add a sweep suite" step. ## Rationale for this change Fixes for Spark's `-0.0` and NaN rules have landed one expression at a time, and nothing shows which paths still differ from Spark or notices when a DataFusion upgrade changes float behavior. This suite runs the edge values through every operator context that hosts a comparison or a float key, and through the expressions that take float input, so the remaining gaps are listed in one place and each is tied to an issue. ## What changes are included in this PR? A new `CometFloatSemanticsSuite`, with 412 cases split across `DOUBLE` and `FLOAT`: - **Edge values:** `0.0`, `-0.0`, a canonical NaN, a sign-bit NaN, `1.0`, `-1.0`, both infinities and `NULL`. Spark's Parquet writer canonicalizes NaN, so the sign-bit NaN is made at query time by negating a stored NaN. That works on every platform. A 9 × 9 table holds every ordered pair. - **Comparisons:** all seven comparison operators in Project, Filter, an aggregate argument, a `FILTER` clause, broadcast hash, shuffled hash, sort-merge and nested loop join conditions, a sort key, `Generate`, a window argument and a window partition key. Array and struct operands are covered too. - **Literal comparisons:** each edge literal on either side, including a constant-folded sign-bit NaN, in Project, Filter, and a data filter pushed into the Parquet reader. - **Keys:** `GROUP BY`, `DISTINCT`, `count(DISTINCT)`, the three equi-join strategies, null-safe, semi and anti joins, `INTERSECT`, `EXCEPT`, `IN` lists and subqueries, `ORDER BY`, window partition and order keys with `rank`/`dense_rank`, hash repartitioning, and nested grouping and sort keys. - **Expressions:** 38 expressions that take float input (math, casts, hashes, `greatest`/`least`, `nanvl`, `IN`, `CASE`, the array functions, map lookups), plus 13 aggregate forms. Aggregates run over every pair of edge values as a two-row group, so `max`/`min` tie-breaking and NaN ordering are both exercised. A case in `knownGaps` is expected to differ from Spark, and it fails once it matches. A fix, or a DataFusion upgrade that changes float behavior, therefore has to remove the entry. The gaps on `main`: | Gap | Issue | | --- | --- | | Comparisons in aggregate arguments, `FILTER`, sort keys, `Generate`, and non-equality join conditions | #6385 (#6447) | | A sign-bit NaN literal in a data filter pushed into the Parquet reader drops rows Spark keeps | #6385 (#6447) | | Ordering and `<=>` comparisons on arrays and structs | #6157 | | Nested `ORDER BY` keys | #5507 | | `hash`, `xxhash64` and shuffle hash partitioning of a sign-bit NaN | #6385 (#6413) | | `min`, `max`, `greatest`, `least`, including window `max`/`min` | #6385 (#6457) | | `array_remove`, `sort_array` | #6385 (#6518) | | `array_distinct`, `array_union` before SPARK-54918 | #5701 | | `collect_set` before Spark 4.2 | #5312 | | `signum(-0.0)` returns `0.0`; Spark returns `-0.0` | #6385, no issue yet | Two of these are not on the epic's list: `signum(-0.0)`, and window `max`/`min`, which may need more than #6457 changes. Should I file issues for them? The suite also turned up a Spark 4.1.3 bug. A single whole-stage-codegen projection holding `x = 0.0`, `0.0 = x`, `x = -0.0`, `-0.0 = x` and the matching NaN and `1.0` comparisons returns `false` for `0.0 = 0.0`, and is right with subexpression elimination off. Smaller subsets are right, and Spark 3.4 and 3.5 are not affected. The literal cases put each predicate in its own `UNION ALL` branch to stay off that path, and a comment says why. The suite is added to the `expressions` group in `pr_build_linux.yml` and `pr_build_macos.yml`. ## How are these changes tested? The change is the suite itself. It passes, with all 412 cases, on every Spark profile: 3.4, 3.5, 4.0, 4.1 and 4.2 (macOS aarch64). On 4.2 the `array_distinct`/`array_union` gap does not apply, because Spark 4.2 includes SPARK-54918, and the gap is gated on the Spark patch version. The `collect_set` gap is likewise gated to before 4.2. Before writing the known gaps, I ran the suite on 4.1 with the list empty and read every mismatch. All 188 were result differences, not errors. The `scalafix`, `scalastyle` and `spotless` checks pass on Spark 3.5, and `dev/ci/check-suites.py` passes. The suite runs in under a minute. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
