andygrove commented on code in PR #7340:
URL: https://github.com/apache/datafusion-comet/pull/7340#discussion_r4429333268
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -312,6 +318,60 @@ fn parquet_convert_array_impl(
Ok(Arc::new(string_array))
}
+ // Unannotated BINARY read as a string. Spark's cast replaces
malformed UTF-8 with
+ // U+FFFD, where Arrow's safe cast below would return NULL. Top-level
columns don't
+ // reach this arm (#7032). The reader keeps a file's ARROW:schema
types, so an Arrow
+ // writer's `LargeBinary` or `BinaryView` arrives as is; Spark's cast
takes only
+ // `Binary`, and converting `LargeBinary` reuses its value buffer.
+ (Binary | LargeBinary | BinaryView, Utf8) => {
+ let array = match from_type {
+ Binary => array,
+ _ => cast_with_options(&array, &Binary, &PARQUET_OPTIONS)?,
+ };
+ if in_map_key {
+ // Spark tells map keys apart by their bytes. Decoding turns
the keys `FF` and
+ // `EF BF BD` into the same U+FFFD, so a lookup of one returns
the other's value
+ // and `map_from_entries` fails with DUPLICATED_MAP_KEY. A key
keeps its bytes
+ // instead, unchecked, as a top-level value does (#7032).
+ let (offsets, values, nulls) =
array.as_binary::<i32>().clone().into_parts();
+ let keys = unsafe { StringArray::new_unchecked(offsets,
values, nulls) };
Review Comment:
@sunchao We already merged the same backport to 1.2 so we probably need to
merge this one too. Is this issue here specific to this backport only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
