kosiew commented on PR #15295:
URL: https://github.com/apache/datafusion/pull/15295#issuecomment-2893829674
I propose to close this large PR and re-implement as:
## PR 1: Extract and test core SchemaAdapter helpers
**Description**
Pull the field‐mapping logic out of `DefaultSchemaAdapter` into two new
helpers and add a simple constructor. Include unit tests for:
- **`can_cast_field`**: succeeds on castable types, errors otherwise
- **`create_field_mapping`**: yields the correct projection & mapping on a
toy schema
- **`SchemaMapping::new`** ctor
**Changed files**
- `datafusion/datasource/src/schema_adapter.rs`
- **New tests**: `datafusion/datasource/src/schema_adapter_tests.rs`
---
## PR 2: Add schema_adapter_factory to ListingTableConfig (with tests)
**Description**
Extend `ListingTableConfig` so users can supply a `schema_adapter_factory`:
- `#[derive(Default)]` on `ListingTableConfig`
- Add field `schema_adapter_factory: Option<Arc<dyn SchemaAdapterFactory>>`
- Builder method `with_schema_adapter_factory(...)`
- Propagate through `new`, `new_with_multi_paths`, and all `to_*_config`
methods
**Unit tests** in `datafusion/core/src/datasource/listing/table_tests.rs`
verify:
- Default config has no factory
- `with_schema_adapter_factory` sets it
- Factory survives a round-trip through `to_scan_config`
**Changed files**
- `datafusion/core/src/datasource/listing/table.rs`
- **New tests**: `datafusion/core/src/datasource/listing/table_tests.rs`
---
## PR 3: Hook FileSourceExt & Parquet preservation (with tests)
**Description**
Allow any `FileSource` to carry a schema-adapter, and ensure Parquet scans
preserve it:
1. **`FileSourceExt`** trait + `with_schema_adapter(...)` impl in `table.rs`
2. In `datasource-parquet/src/file_format.rs`, add
`preserve_conf_schema_adapter_factory` and call it in `scan()`
3. **Unit tests** in `datasource-parquet/src/file_format_tests.rs` confirm:
- A `ParquetSource` with an adapter remains carrying that factory after
`scan()`
**Changed files**
- `datafusion/core/src/datasource/listing/table.rs`
- `datafusion/datasource-parquet/src/file_format.rs`
- **New tests**: `datafusion/datasource-parquet/src/file_format_tests.rs`
---
## PR 4: Nested-struct SchemaAdapter implementation & re-exports (with tests)
**Description**
Ship the full nested-struct adapter plus public re-exports:
- `nested_schema_adapter.rs` with `NestedStructSchemaAdapterFactory`,
`NestedStructSchemaAdapter`, mapping logic, helpers
- Export via:
- `datafusion/datasource/src/mod.rs`
- `datafusion/core/src/datasource/mod.rs`
- **Comprehensive unit tests** (already present) cover adapter selection,
struct-field evolution, batch mapping, and stats mapping
**Changed files**
- `datafusion/datasource/src/nested_schema_adapter.rs`
- `datafusion/datasource/src/mod.rs`
- `datafusion/core/src/datasource/mod.rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
