cj-zhukov opened a new issue, #26863:
URL: https://github.com/apache/datafusion/issues/26863
### Is your feature request related to a problem or challenge?
`DataFrame::from_columns` marks every column as nullable, even when the
array passed in has no nulls:
```rust
let id: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3]));
let df = DataFrame::from_columns([("id", id)])?;
assert!(df.schema().field_with_unqualified_name("id")?.is_nullable()); //
true today
```
This loses information the caller already has. The schema does not reflect
the data when it is written out or shared with other systems. It also differs
from arrow’s own `RecordBatch::try_from_iter`, which derives nullability from
the arrays.
### Describe the solution you'd like
Build each field with `array.is_nullable()` instead of `true`. A column is
then non-nullable when its array is known to contain no nulls. Arrays that may
contain logical nulls (for example a dictionary with nullable values) stay
nullable, so the resulting `RecordBatch` is always valid. The doc comment
should mention that nullability is derived from the array.
### Describe alternatives you've considered
- Keep every column nullable (current behavior). Simple and predictable, but
the schema is always weaker than the data, and callers who want `NOT NULL` have
to build the `RecordBatch` and call `ctx.read_batch` themselves.
- Use `array.null_count() > 0` instead of `is_nullable()` in
`RecordBatch::try_from_iter`
### Additional context
- This is a behavior change, not an API change: the signature stays the
same, but schemas produced by `from_columns` can now differ depending on
whether the arrays contain nulls. Code that compares the result against a
hand-built all-nullable schema may need updating.
- `dataframe!` macro goes through `from_columns`, so it gets the same
behavior.
- Matches `RecordBatch::try_from_iter`, which also sets nullability from the
data.
### Willingness to contribute
I plan to contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
