NoahKusaba opened a new issue, #2881: URL: https://github.com/apache/datafusion-ballista/issues/2881
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** `ballista-iceberg` (#2638) pins each catalog-backed Iceberg table to one version when a query looks it up, in `PinningSchema::table`, so the plan can be encoded without I/O. #2784 extends that to views by re-pinning a view's stored plan on each lookup. Choosing the version at lookup still leaves gaps that single-process DataFusion doesn't have, since there `IcebergCatalogTableProvider` loads its table in `scan()`: - A reused `DataFrame`, or a Flight SQL prepared statement run more than once, reads the version from when it was planned. Neither looks its tables up again. - A query that reads a table both through a view and directly looks it up twice, so a commit landing between the two lookups gives them different versions. - Lookups do catalog I/O, so anything that only needs a table's name, type or schema (`DROP`, `DESCRIBE`, `information_schema.columns`) loads it, and fails if it can't be loaded. #2784 works around this for views with `RepinnedView`, and the same problem for base tables is tracked separately. **Describe the solution you'd like** Pin when a query runs: in the async `QueryPlanner::create_physical_plan`, walk the whole plan once, load each distinct catalog-backed table once, and swap in a pinned version of each scan. Lookups would then do no I/O, every scan of a table in a query would read one version, and each run of a reused `DataFrame` or prepared statement would read the latest version. The re-pinning in `PinningSchema::table` and `RepinnedView` would go away. This needs a hook in Ballista's core first. `SessionStateExt::upgrade_for_ballista` (`ballista/core/src/extension.rs`) only lets `ballista_query_planner()` replace `BallistaQueryPlanner`, so an integration can't wrap it. Either a composable query planner, or a hook that runs on the logical plan before `DistributedQueryExec` is built, would do. The scheduler plans Flight SQL queries itself, so it needs the same hook. **Describe alternatives you've considered** - Pin at lookup and re-pin views (#2638, #2784): what exists now, with the gaps above. - Re-plan a view from its SQL: `SchemaProvider::table` gets no session state to plan with, and a view created from a `DataFrame` has no SQL (#2726). **Additional context** Follow-up to #2638 and #2784. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
