jayzhan211 opened a new pull request, #24522:
URL: https://github.com/apache/datafusion/pull/24522
## Rationale for this change
`build_read_plan_with_cast_clipping` previously tracked projection decisions
across several overlapping collections. This made it difficult to understand
whether each root column required a full or partial read and required
additional
work to combine cast and `get_field` projections.
This PR simplifies that planning path while preserving its conservative
fallback behavior.
## What changes are included in this PR?
- Represent each projected root with one explicit state: a full read or a set
of selected leaf offsets.
- Merge cast and `get_field` leaf requirements into the same per-root state.
- Preserve full-root reads whenever partial projection cannot be proven safe.
- Build the projected schema and leaf mask in one ordered finalization loop.
- Avoid an additional struct-leaf resolution pass and several temporary
collections.
- Remove the final sort and deduplication because root ordering, descriptor
ordering, and `BTreeSet` offsets already produce sorted unique leaf
indices.
- Add comments explaining the ordering and fallback invariants.
## Are these changes tested?
Yes. Existing tests cover full and partial projections, repeated and
overlapping
casts, cast and `get_field` combinations, fallback behavior, and stale column
indices.
The following checks passed:
- `cargo fmt --all --check`
- `cargo clippy -p datafusion-datasource-parquet --lib -- -D warnings`
- `cargo test -p datafusion-datasource-parquet --lib`
- `cargo test -p datafusion-sqllogictest --test sqllogictests --
parquet_nested_schema_pruning`
## Are there any user-facing changes?
No. This is an internal performance and readability improvement with no
public
API or intended query behavior changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
