dd-david-levin opened a new pull request, #25197:
URL: https://github.com/apache/datafusion/pull/25197
## Which issue does this PR close?
- Follow-up to #20646. This is a draft for design discussion; no issue has
been filed yet.
## Rationale for this change
The Substrait consumer currently rejects a `ReadRel` when a table provider
reports a narrower numeric type than the logical type declared by the Substrait
`base_schema`, even when promoting the provider type is lossless.
A concrete case is a table field with type:
```text
Struct("period_unit": Utf8, "period_value": Int32)
```
and a Substrait read schema with type:
```text
Struct("period_unit": Utf8, "period_value": Int64)
```
The read projection already casts fields to the Substrait schema, but the
compatibility check rejects the plan before that projection can be built. This
occurs with producers that normalize integer widths in their logical type
system while table providers retain narrower storage types.
#20646 proposed allowing top-level numeric widening. This draft revisits
that proposal with the concrete nested-struct case and recursive validation. I
am opening it as a draft because the earlier discussion questioned whether such
plans should instead be rejected or fixed by producers.
## What changes are included in this PR?
- Accept lossless signed-integer, unsigned-integer, and floating-point
widening when validating read schemas.
- Apply the same compatibility rule recursively to `Struct` children.
- Continue requiring struct arity, field order, and field names to match.
- Continue rejecting numeric narrowing and unrelated type changes.
## What is the testing strategy for this PR?
Added unit coverage for:
- top-level `Int32` to `Int64` widening;
- nested-struct `Int32` to `Int64` widening;
- rejection of nested `Int64` to `Int32` narrowing.
Validation run:
```text
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
RUST_BACKTRACE=1 cargo test --profile ci \
--exclude datafusion-examples --exclude datafusion-benchmarks --exclude
datafusion-cli \
--workspace --lib --tests --bins \
--features
avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption
```
## Are there any user-facing changes?
Substrait `ReadRel` plans may resolve tables whose numeric fields are
narrower than the corresponding declared Substrait fields when promotion is
lossless, including fields nested inside structs. The resulting projection
retains the types declared by the Substrait plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
