mbutrovich opened a new issue, #118: URL: https://github.com/apache/datafusion-iceberg/issues/118
`IcebergDataSource` stores `snapshot_id: Option<i64>`, and each consumer resolves it against the table metadata again: - `IcebergStaticTableProvider::try_new_from_table_snapshot` looks up the snapshot, returns an error if it's missing, and then keeps only the id ([static_table_provider.rs](https://github.com/apache/datafusion-iceberg/blob/a7e5f555970bf5ee61c6f47d3815d503ff837b2a/crates/datafusion/src/table_provider/static_table_provider.rs#L74-L94)). - `get_batch_stream` passes the id to iceberg's `TableScanBuilder`, which looks it up again at execution and fails there if it's missing ([scan.rs](https://github.com/apache/datafusion-iceberg/blob/a7e5f555970bf5ee61c6f47d3815d503ff837b2a/crates/datafusion/src/read/scan.rs#L278-L281), [iceberg-rust `TableScanBuilder::build`](https://github.com/apache/iceberg-rust/blob/5b82ab99aeaf50bf4f5b3b776ac9cd6380a5f3e3/crates/iceberg/src/scan/mod.rs#L285-L296)). - `estimated_num_rows`, added in #114, looks it up a third time for `partition_statistics`. It has to handle a missing id by reporting `Precision::Absent`, because the builder accepts any id. Could `IcebergDataSourceBuilder::build` resolve the id, or the current snapshot when it's `None`, into an `Option<SnapshotRef>`, and return an error for an id that doesn't exist? Each consumer would then read the stored snapshot. A missing snapshot would fail at planning instead of at execution, and the `Absent` branch in `estimated_num_rows` would go away. `IcebergDataSource::snapshot_id()` is public. It returns `None` today to mean "the current snapshot". If the source stores the resolved snapshot, `None` would mean "the table has no snapshot". The issue should settle which meaning the getter keeps before the change ships. #67 restructures the builder's input (`snapshot_id` moves into its table input), so this should land after #67 or be rebased onto it. This came up in review of #114: https://github.com/apache/datafusion-iceberg/pull/114#discussion_r4607108117 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
