snmvaughan opened a new pull request, #7266: URL: https://github.com/apache/datafusion-comet/pull/7266
## Which issue does this PR close? Backport of #7171 (squash commit 1dec4c058) to `branch-1.1`, for #7170. ## Rationale for this change Spark 4.0.5, 4.1.4 and 4.2.1, now in release candidates, change two results that Comet reproduces natively: - When `stop - start` overflows a long but the sequence is short, `sequence` returns its elements, where earlier releases raise `INTERNAL_ERROR` (SPARK-58821). - A string zero with a large exponent, such as `'0e39'`, casts to 0 instead of NULL, or `NUMERIC_OUT_OF_SUPPORTED_RANGE` under ANSI (SPARK-60118). Without this change, Comet 1.1 on those releases raises the internal error and returns NULL where Spark doesn't. See #7170 and #7171. ## What changes are included in this PR? `git cherry-pick -x` of 1dec4c058, adapted because `branch-1.1` predates parts of `main` next to #7171's lines. The behavior matches `main`: - `sequence.rs`: `branch-1.1`'s kernel takes the start, stop and step arrays and a conversion closure, and its `sequence_length` flags the overflow itself, so the `OverflowedLength` choice is threaded through that code. `recomputed_length_after_overflow` also checks that a length over the limit is still an error with `spark_sequence_recomputed`, which `main`'s length grid covers. - `string.rs`: the `zero_any_exponent` change and its test. The SPARK-60119 comment rewording is dropped, since `branch-1.1` doesn't have the scale-adjustment comments it edits. - `planner.rs`: `branch-1.1`'s `SparkCastOptions::new_with_version` still takes `allow_incompat`. - The contributor-guide audit entries are dropped, since that guide isn't published from release branches and they describe `main`'s code. `SparkFixes`' scaladoc doesn't mention `ArraySetSupport.normalizesFloats`, which `branch-1.1` doesn't have. - Import lists keep `branch-1.1`'s names plus the ones #7171 adds. Backports to other branches: `branch-1.2`, in a separate PR. ## How are these changes tested? The tests come with the cherry-pick, plus the extra length-limit check above. On this branch, all 978 `datafusion-comet-spark-expr` unit tests pass, `cargo fmt --check`, `cargo clippy --all-targets --workspace -- -D warnings` and spotless are clean, and the Scala code compiles, with `SparkFixesSuite` passing. I haven't run the other Scala suites locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
