andygrove opened a new issue, #41: URL: https://github.com/apache/datafusion-iceberg/issues/41
### Describe the bug `IcebergMetadataTableProvider::scan` ignores `projection` ([metadata_table.rs#L61-L69](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/table/metadata_table.rs#L61-L69)). `IcebergMetadataScan` always declares and returns the full metadata table schema ([metadata_scan.rs#L37-L48](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/physical_plan/metadata_scan.rs#L37-L48)), but DataFusion expects the plan to produce the projected schema. ### To Reproduce On a table `t` with two snapshots: - `SELECT operation FROM t$snapshots` returns all six columns (`committed_at`, `snapshot_id`, `parent_id`, `operation`, `manifest_list`, `summary`). - `SELECT count(*) FROM t$snapshots` fails with: ``` Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences: - Different number of fields: (physical) 6 vs (logical) 0. ``` - `SELECT operation, snapshot_id FROM t$snapshots` fails with: ``` Internal error: Assertion failed: col.name() == matching_name: Input field name snapshot_id does not match with the projection expression operation. ``` Only `SELECT *` works. ### Expected behavior Projected and aggregate queries over metadata tables return the requested columns. ### Additional context This was reported upstream as apache/iceberg-rust#2819, before the integration moved here. Applying the projection to each batch, and to the schema in the plan properties, would fix it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
