Hi Anton, Thank you for putting this proposal together and for the thoughtful revisions following the discussion.
I have one small suggestion on metadata caching. The gateway already reuses the Rust client's cached TableInfo and refreshes it when needed. Could we build on these client-side facilities for FIP-50, extending them where necessary, rather than maintaining a separate long-lived metadata cache in fluss-datafusion? This could let SQL and direct access reuse the same metadata caching, refresh and invalidation APIs.Protocol-specific decoder caches, DataFusion providers and query-scoped state could still remain in their respective adapters. Best regards, Junbo Wang > On 17 Aug 2026, at 20:31, Jim Hu <[email protected]> wrote: > > Hi Leonard and Anton, > > Thanks for the suggestions. I agree that non-tiered PK reads should be a > follow-up and need not block FIP-50. FIP-48 can remain lake-scoped. > > In Java, the KV and lake scanners handle their source-specific input > collection separately, while sharing the final snapshot/changelog > reconciliation and delete handling through SortMergeReader in fluss-client. > We can follow the same boundary in Rust without adopting the sort-merge > algorithm itself. The existing hash-overlay logic, including tail folding, > tombstone handling, base-image reconciliation, and survivor emission, can be > made source-neutral and moved from fluss-lake into the core crate. > > The lake path can use it with a lake snapshot, while a future non-tiered path > can use the same component with a KV snapshot from ScanKv. This avoids > duplicating merge and delete semantics and provides the foundation for the > engine-neutral current-view contract Leonard suggested. > > Thanks Anton for the clarification and for the detailed update! > > Best, > Jim > >
