Xuanwo opened a new pull request, #27502:
URL: https://github.com/apache/datafusion/pull/27502

   ## Which issue does this PR close?
   
   Partially addresses #21982: this draft covers `make_array` and 
`named_struct`, together with the argument-coercion behavior needed to keep 
their schemas consistent. The issue also covers other constructors and 
aggregates, so this PR does not close it.
   
   Earlier attempts #21984 and #22176 are closed without merging. This draft 
has a smaller constructor scope and also addresses metadata consistency across 
implicit argument conversion.
   
   ## Rationale for this change
   
   Wrapping a column in an array or named struct currently drops its 
child-field metadata, including Arrow extension-type annotations. A consumer 
can therefore receive a different logical type after a SQL projection even 
though the underlying values were only wrapped in a container.
   
   Preserving metadata also requires array arguments to share an exact item 
type. Structs with identical child storage types can still differ in child 
metadata or nullability; passing those different types to Arrow's array 
concatenation can panic. The common type must be reflected in both the planned 
schema and the actual input arrays.
   
   For example, these constructors should preserve the declared child fields 
and execute consistently when `name` and `l_name` are Utf8 columns carrying 
different ordinary field metadata:
   
   ```sql
   SELECT make_array(
     named_struct('v', name),
     named_struct('v', l_name)
   )
   FROM table_with_metadata;
   ```
   
   ## What changes are included in this PR?
   
   - Preserve value fields in `named_struct`; derive `make_array`'s item field 
from the coerced argument fields and use that field at execution time.
   - Apply type-only cast metadata semantics in shared function argument 
coercion and successful implicit literal materialization: preserve ordinary 
metadata, remove both extension keys when the argument type changes, and retain 
the complete field when its type is unchanged. Logical and physical type-only 
casts use the same helper.
   - Coerce struct array arguments to one exact item type. Child nullability is 
combined across inputs; child metadata currently comes from the first coerced 
struct, following existing list-coercion precedence.
   
   The only new public item is `FieldMetadata::without_extension_type`, marked 
`#[doc(hidden)]` and documented as internal to the workspace. Existing 
supported public signatures are unchanged.
   
   ### Draft design questions
   
   The first-input policy for struct child metadata needs further investigation 
before this is ready for review. In particular, nested extension-name conflicts 
are not explicitly reconciled, and the underlying struct coercion helper can 
retain a child's extension annotation when that child's storage type changes. 
The ordinary-metadata regression tests below do not establish correctness for 
those nested extension cases.
   
   Implicit literal conversion now drops extension annotations when its storage 
type changes, matching column conversion. This is an observable semantic change 
and is called out explicitly for review.
   
   ## What is the testing strategy for this PR?
   
   Public SQL/DataFrame tests in `datafusion/core/tests/sql/select.rs` check 
both planned and executed types, values and offsets, NULL inputs, parameter 
metadata, type promotion, extension-name conflicts, and struct arguments with 
different metadata/nullability in both argument orders. A focused coercion test 
checks that only converted arguments lose extension annotations.
   
   Local validation on commit `9beb06c145792b4ec772b0365a7f42d0b2603490`, based 
on `120049489b5b74a3c8d942e5a97a1b7dbeec0ac2`:
   
   - Formatting and `cargo clippy --all-targets --all-features -- -D warnings` 
passed, using the CI profile with LTO disabled.
   - Required extended workspace coverage completed across all 62 test 
executables, including 307 SQLLogicTest files. The initial run hit `EMFILE` in 
`test_sort_10k_mem` with a soft file-descriptor limit of 1024. Raising that 
limit to 65535 and running the failed case and unreached targets completed the 
coverage; successful cases from the same source were retained.
   - The original metadata-difference SQL probe passes with the rebuilt 
SQLLogicTest executable.
   
   ### Performance observations
   
   Local `release-nonlto` measurements compare the commit above with its 
unmodified base, using serial ABBA order, 12 pinned CPUs, and 30 samples per 
case per round. The SQL workloads use 65,536 rows in 16 batches, with both 
uniform non-null inputs and irregular values/nulls/string lengths.
   
   Planning `make_array(a, a)` with a metadata-bearing Int32 column increased 
from 21.161 to 22.010 microseconds (+4.01%). Planning `make_array(a, b)` with 
Int32-to-Int64 promotion increased from 21.995 to 23.121 microseconds (+5.12%). 
Full SQL-to-collect measurements across the constructor workloads changed by 
-2.07% to +2.05%. These observations show a planning cost; they are not a claim 
of an overall performance improvement.
   
   ## Are there any user-facing changes?
   
   `named_struct` and `make_array` retain child-field metadata through planning 
and execution. Struct array inputs with different ordinary metadata/nullability 
are normalized to one type. Implicit literal storage conversion removes 
extension annotations in the same way as column conversion, while preserving 
ordinary metadata.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to