andygrove commented on code in PR #5769:
URL: https://github.com/apache/datafusion-comet/pull/5769#discussion_r3959014204
##########
native/spark-expr/src/struct_funcs/get_struct_field.rs:
##########
@@ -62,23 +62,11 @@ impl GetStructField {
/// Extract field `ordinal` from a struct array, propagating the parent
struct's null mask.
///
- /// Spark semantics: a field of a NULL struct is NULL. Arrow stores a
StructArray's child
- /// arrays with their own validity, INDEPENDENT of the parent struct's
null buffer -- so the
- /// raw child value at a row where the struct itself is null can be
non-null (e.g. parquet
- /// files where a logically-null struct column still has a populated child
buffer). Returning
- /// the child column verbatim then makes `isnotnull(struct.field)` wrongly
true for a null
- /// struct. Union the struct's null mask into the child's (null where the
struct is null OR
- /// the child is null).
+ /// Spark semantics: a field of a NULL struct is NULL. Without the union,
+ /// `isnotnull(struct.field)` reports true for a null struct. See
+ /// `datafusion_comet_common::struct_nulls` for why the child can hold a
value there.
fn project_field(struct_array: &StructArray, ordinal: usize) ->
DataFusionResult<ArrayRef> {
Review Comment:
seems like this function could just be removed now and the callers can be
updated to call `child_with_parent_nulls` directly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
