sunchao opened a new pull request, #6170: URL: https://github.com/apache/datafusion-comet/pull/6170
## Which issue does this PR close? Part of #4496. Depends on #5403; this draft is stacked on that PR and should be rebased after it lands. Review the follow-up separately in [commit 821dbf1be](https://github.com/sunchao/arrow-datafusion-comet/commit/821dbf1bee38e76b50515f4c0c7a4a2a3e9b7e0c). ## Rationale for this change #5403 uses Spark's JVM dispatcher for collated `array_min`/`array_max`. This follow-up executes the UTF8 collation family in Rust, including comparisons inside arrays and structs. For example, `array_min` of `['a', 'B']` under `UTF8_LCASE` returns the original `'a'` without invoking Spark's comparator. ## What changes are included in this PR? - Carry string-leaf collations into the existing native extrema comparator. Support `UTF8_BINARY`, `UTF8_BINARY_RTRIM`, `UTF8_LCASE`, and `UTF8_LCASE_RTRIM`, retaining the first original value on ties. - Match Spark's Unicode 16/17 lowercase rules, including dotted I and final sigma, using a pinned, generated mapping table. The generator records source commits and checksums; no ICU dependency is added to native builds. - Allow byte-preserving string casts that assign collations to run natively, while preserving Spark's CHAR/VARCHAR length checks. Keep locale-based ICU collations on the existing dispatcher path. - Update the existing focused JVM/SQL coverage and add two native tests covering string layouts, original winners, and version-specific comparison rules. The repository's `implement-comet-expression` skill guided the implementation and source audit. ## How are these changes tested? - Native extrema tests: 11 passed. - Spark 4.1: 5 tests passed across the focused `CometArrayExpressionSuite` extrema cases, `CometSqlFileTestSuite array_extrema_collation`, and a CHAR-padding regression, with a JNI library built from this branch. Native-path tests disable the JVM dispatcher and check nested mixed collations and output schema. - Exhaustive comparison against the pinned Unicode sources: 2,224,128 versioned scalar checks, with zero differences. Generator output is reproducible. - `cargo clippy --workspace --all-targets -- -D warnings`, rustfmt, Spotless, ScalaStyle, Apache RAT, Markdown formatting, and `git diff --check` pass. All Spark profiles and the Spark 4.1 SQL suite are requested through CI labels. Their hosted verdicts are pending. No end-to-end performance result is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
