manuzhang opened a new pull request, #7340: URL: https://github.com/apache/datafusion-comet/pull/7340
## Which issue does this PR close? Backport of #7106 to `branch-1.1`. Part of #7032 (the nested values) and #4764. ## Rationale for this change When a Parquet BINARY column without the UTF8 annotation is read as a string, through a read schema or `spark.sql.parquet.binaryAsString=true`, the native scan converts nested values (struct fields, list elements, map keys and values) in `parquet_convert_array`. That fell through to Arrow's safe cast, which turns invalid UTF-8 into NULL, so those values silently became NULL where Spark keeps the bytes. `branch-1.1` has this bug. See #7106 for the full rationale. The fix also goes to `branch-1.2` in #7339, opened together with this one. Per the backporting guide, this one should not merge before that one is ready. ## What changes are included in this PR? The cherry-pick of 621b9ccdb4658ad2671e4e88e5c51da67e3c8fbc from `main`. `branch-1.1` differs from `main` in several places the source commit touches, so these parts were applied by hand: - `native/core/src/parquet/parquet_support.rs`: on this branch `parquet_convert_array_impl`, `parquet_convert_struct_to_struct` and `parquet_convert_map_to_map` return a bare `ArrayRef`, not the `(ArrayRef, bool)` field-overlap pair that #7085 introduced on `main`. The new `in_map_key` parameter is threaded through the branch's signatures, and the new `Binary`/`LargeBinary`/`BinaryView` and `Dictionary` arms return a bare array. The helper functions `referenced_dictionary_values` and `decoded_bytes` and the four new unit tests are taken verbatim from the source commit. The `SparkError` import conflict was resolved the same way as on `branch-1.2`. - `native/core/src/parquet/parquet_exec.rs`: the new test `nested_arrow_binary_types_read_as_string_decode_invalid_utf8` is taken from the source commit, with the three trailing datetime-rebase arguments dropped from its `init_datasource_exec` call, because that function doesn't take them on this branch. - `native/spark-expr/src/conversion_funcs/cast.rs`: only the import line conflicted. `main` imports `format_decimal_str` there and this branch doesn't; the backport adds only the `arrow::buffer` imports the fix needs. The rest of the file's changes applied cleanly. - On this branch `SparkCastOptions::new`, `SparkCastOptions::new_without_timezone` and `SparkParquetOptions::new` still take the `allow_incompat` flag that `main` has since removed, so the backported code and tests pass `false` to them. - `docs/source/user-guide/latest/compatibility/index.md`: this branch's "Strings with non-UTF-8 bytes" section predates the per-source list that `main` has, so the source commit's new bullet is folded into the existing paragraph as a sentence: nested BINARY values read as strings are now decoded, top-level values and map keys are not checked, with a link to #7032. - `docs/source/user-guide/latest/parquet.md` does not exist on this branch, so its changes (the #7032 known-issue entry and the #7254 dictionary limit) are dropped. ## How are these changes tested? The tests from #7106: the Rust unit tests in `parquet_support.rs`, `parquet_exec.rs` and `cast.rs`, and the new `ParquetReadSuite` test "native scan decodes invalid UTF-8 in nested BINARY read as string". Locally on this branch: - `cargo check --all-targets`, `cargo clippy --all-targets --workspace -- -D warnings` and `cargo fmt --check` in `native/` pass. - The `parquet_support` and `parquet_exec` tests in `datafusion-comet` (22 tests) and the `cast` tests in `datafusion-comet-spark-expr` (95 tests) pass. - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.parquet.ParquetReadV1Suite native scan decodes"` passes after `make core`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
