sam-1112 opened a new pull request, #5839:
URL: https://github.com/apache/datafusion-comet/pull/5839
## Which issue does this PR close?
Closes #5743.
## Rationale for this change
`slice` crashed when its input array had a non-nullable element field. Spark
declares expressions such as `split` and `sequence` as `ArrayType(...,
containsNull = false)`, so they can produce a non-nullable child field inside
the native plan, after the scan boundary.
This differs from #4789, where the non-nullable field entered through
`CometLocalTableScanExec` and could be normalized at that boundary. The
reproducer for #5743 uses a native Parquet scan; its non-nullable field is
produced later by an expression, so the #4789 fix does not cover it.
The planner fix is already present on `main` through #5766: `CometSlice` no
longer serializes an explicit return type. Native planning therefore uses
`SparkArraySlice.return_field_from_args`, whose return field follows the actual
input list field.
This PR adds the missing regression coverage for that path.
## What changes are included in this PR?
No production-code changes.
- JVM regression coverage for:
- `slice(split(...))`
- `slice(sequence(...))`
- `slice(concat(split(...), array(...)))`
- The JVM cases use a native Parquet scan and `checkSparkAnswerAndOperator`,
verifying both Spark-compatible results and native execution.
- A Rust unit test verifies that `slice_list` preserves the input list's
inner field, including its name, nullability, and metadata.
## How are these changes tested?
- `cargo test -p datafusion-comet-spark-expr array_slice`
- `make core`
- `./mvnw test -Dtest=none
-Dsuites="org.apache.comet.CometArrayExpressionSuite slice over
expression-produced non-null element arrays"`
- `./mvnw test -Dtest=none
-Dsuites="org.apache.comet.CometArrayExpressionSuite"`
- `make release`
- `./mvnw test -Prelease -Dtest=none
-Dsuites="org.apache.comet.CometArrayExpressionSuite slice over
expression-produced non-null element arrays"`
The targeted JVM test was run with the default Spark 4.1.3 profile, matching
the original reproduction environment. Both debug and release builds were
exercised because #5743 failed at different validation points in those
configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
