sunchao opened a new pull request, #5846: URL: https://github.com/apache/datafusion-comet/pull/5846
## Which issue does this PR close? Follow-up to #5045. This addresses additional map-constructor failures exposed by composed map expressions. ## Rationale for this change `map_from_arrays` can silently pair values from different rows when the two input columns have equal flattened lengths but different lengths in individual rows. For example, keys `[[1], [2, 3, 4]]` and values `[[10, 20], [30, 40]]` must fail, but the current native constructor accepts them and assigns values across row boundaries. The null guard also evaluates the values expression for rows whose keys are null when another row in the batch has non-null keys. Under ANSI mode, this can raise a division-by-zero error that Spark skips. ## What changes are included in this PR? - Add a small adapter around the existing DataFusion map constructor to validate every non-null row's key/value lengths and batch sizes. Mixed scalar/batch operands are expanded for validation; scalar-only construction retains its existing path. - Replace the combined `AND` null check with nested `CASE` guards so a null keys array suppresses evaluation of the values expression for that row. - Add focused native and Spark regression coverage for both failures, scalar/batch inputs, null rows and values, empty batches, and sliced arrays. This preserves the existing map constructor, map types, and admission policy. The separate nondeterministic-child evaluation issue #5781 is unchanged. ## How are these changes tested? - `cargo fmt --all -- --check` and `git diff --check` passed. - Full Maven reactor test compilation, Spotless, and scalastyle passed on Spark 4.1 with JDK 17: `./mvnw test-compile -DskipTests`. - The focused native test command (`cargo test --locked -p datafusion-comet-spark-expr --lib map_funcs::map_from_arrays`) was attempted but stopped before compilation because the managed dependency registry does not provide the locked `aws-smithy-runtime-api` 1.16.0 dependency. The lockfile was not changed. - Native tests and Spark runtime tests have not run locally. The Spark regressions are in `CometMapExpressionSuite` and cover JVM codegen dispatch disabled and enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
