Hi Junbo, Agreed, I'll change it. The client's Metadata and Cluster already cache TableInfo, partition ids and bucket locations, with check_and_update_table_metadata and evict_table_metadata for refresh and invalidation, and the gateway's ConnectionCache keys connections by effective user. So a second cache in fluss-datafusion duplicates that, and my per-principal keying is redundant as metadata is already per-connection.
One gap: nothing caches list_databases or list_tables, and DataFusion's table_names and table_exist are synchronous, so the provider needs a name list without awaiting. By your reasoning that belongs in the client, so REST listing and SHOW TABLES share one path. Reasonable as a small addition. Thank you for flagging this, updated FIP -- Anton ср, 9 сент. 2026 г. в 13:59, Junbo Wang <[email protected]>: > Hi Anton, > > Thank you for putting this proposal together and for the thoughtful > revisions following the discussion. > > I have one small suggestion on metadata caching. The gateway already > reuses the Rust client's cached TableInfo and refreshes it when needed. > Could we build on these client-side facilities for FIP-50, extending them > where necessary, rather than maintaining a separate long-lived metadata > cache in fluss-datafusion? > > This could let SQL and direct access reuse the same metadata caching, > refresh and invalidation APIs.Protocol-specific decoder caches, DataFusion > providers and query-scoped state could still remain in their respective > adapters. > > > > Best regards, > Junbo Wang > > > On 17 Aug 2026, at 20:31, Jim Hu <[email protected]> wrote: > > > > Hi Leonard and Anton, > > > > Thanks for the suggestions. I agree that non-tiered PK reads should be a > follow-up and need not block FIP-50. FIP-48 can remain lake-scoped. > > > > In Java, the KV and lake scanners handle their source-specific input > collection separately, while sharing the final snapshot/changelog > reconciliation and delete handling through SortMergeReader in fluss-client. > We can follow the same boundary in Rust without adopting the sort-merge > algorithm itself. The existing hash-overlay logic, including tail folding, > tombstone handling, base-image reconciliation, and survivor emission, can > be made source-neutral and moved from fluss-lake into the core crate. > > > > The lake path can use it with a lake snapshot, while a future non-tiered > path can use the same component with a KV snapshot from ScanKv. This avoids > duplicating merge and delete semantics and provides the foundation for the > engine-neutral current-view contract Leonard suggested. > > > > Thanks Anton for the clarification and for the detailed update! > > > > Best, > > Jim > > > > > >
