manuzhang opened a new pull request, #7339: URL: https://github.com/apache/datafusion-comet/pull/7339
## Which issue does this PR close? Backport of #7106 to `branch-1.2`. Part of #7032 (the nested values) and #4764. ## Rationale for this change When a Parquet BINARY column without the UTF8 annotation is read as a string, through a read schema or `spark.sql.parquet.binaryAsString=true`, the native scan converts nested values (struct fields, list elements, map keys and values) in `parquet_convert_array`. That fell through to Arrow's safe cast, which turns invalid UTF-8 into NULL, so those values silently became NULL where Spark keeps the bytes. `branch-1.2` has this bug. See #7106 for the full rationale. The fix also goes to `branch-1.1` in a separate pull request, opened together with this one. ## What changes are included in this PR? The cherry-pick of 621b9ccdb4658ad2671e4e88e5c51da67e3c8fbc from `main`, with one adaptation: - `native/core/src/parquet/parquet_support.rs`: the import block conflicted because `main` also imports `SparkError` there (added after the 1.2 cut by the timestamp-overflow change). The backport only takes the new `spark_cast` and `SparkCastOptions` imports and leaves `SparkError` out, since nothing on this branch uses it. Everything else, including the docs changes to `compatibility/index.md` and `parquet.md`, applied cleanly. ## How are these changes tested? The tests from #7106: the Rust unit tests in `parquet_support.rs`, `parquet_exec.rs` and `cast.rs`, and the new `ParquetReadSuite` test "native scan decodes invalid UTF-8 in nested BINARY read as string". Locally on this branch: - `cargo check --all-targets`, `cargo clippy --all-targets --workspace -- -D warnings` and `cargo fmt --check` in `native/` pass. - The `parquet_support` and `parquet_exec` tests in `datafusion-comet` (28 tests) and the `cast` tests in `datafusion-comet-spark-expr` (111 tests) pass. - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadV1Suite native scan decodes"` passes after `make core`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
