NoahKusaba opened a new pull request, #2784: URL: https://github.com/apache/datafusion-ballista/pull/2784
# Which issue does this PR close? Closes #2726. **Stacked on #2638**, so draft until it merges. Only the last commit, "Re-pin a view's Iceberg tables each time a query names it", is new; the rest is #2638. Split out at andygrove's request so the plan rewriting gets its own review. # Rationale for this change #2638 pins each catalog-backed Iceberg table to one version when a query looks it up. DataFusion stores a view as the plan it was created with, so a view over an Iceberg table keeps reading the version from when the view was created. In single-process DataFusion, the same view reads new commits on every run. # What changes are included in this PR? When a query names a view, the pinning schema provider re-pins the view's stored plan: every scan of a pinned catalog-backed table, including scans inside subqueries, is replaced with the version its catalog serves now. Each distinct table is loaded once, and the loads run concurrently. `PinnedTable` keeps the `IcebergCatalogTableProvider` it was pinned from (never sent between nodes), so it can be pinned again. It re-pins the stored plan rather than re-planning the view's SQL. `SchemaProvider::table` gets no session state to plan with, and a view created from a `DataFrame` has no SQL. Nested views need no recursion, since DataFusion inlines a view into any plan built over it. Limitations: - A reused `DataFrame` or Flight SQL prepared statement still reads the version from when it was planned. Neither looks its tables up again. - A query that reads a table both through a view and directly looks it up twice, so a commit landing between the two lookups could give them different versions. Closing that needs a per-query lookup cache in DataFusion, as Spark has. # Are these changes tested? Yes. `should_read_a_views_latest_version` (standalone Ballista over a memory catalog) creates a view over a table and a second view over the first with a subquery on the table, commits a row, and checks that both views return it. The test fails without the re-pinning and without the subquery handling. # Are there any user-facing changes? Yes: views over Iceberg tables now read the latest version. The docs' limitation for views is narrowed to reused `DataFrame`s and prepared statements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
