Hi everyone,
Thanks to Junbo (@beryllw) for proposing a design for union read (lake + log) support in the fluss-rust client. I've talked with Boyu offline — since he's currently focused on the Gateway work, I'd like to take over this proposal and drive it forward. < https://docs.google.com/document/d/1FE56807gsaYxiDMKAuqKavnvJrAmlzblDNnFNZM7msE/edit?tab=t.0 > *Motivation* Fluss already supports union read (lake snapshot + real-time log tail) on the Java side, so lake-enabled tables can be read as a single unified result. The fluss-rust client, however, can currently only read the log side — it has no lake read yet, so a lake-enabled table cannot be read completely from Rust. This also blocks Arrow-native Rust engines such as DataFusion from consuming these tables as a source. This proposal brings union read to the fluss-rust client as an engine-agnostic, Arrow-native layer, so the Rust client can read lake-enabled tables as one unified result, consistent with the Java semantics. As a first step it focuses on bounded batch read and Paimon; streaming union read and other lake formats can be added later as extensions. *Proposal Summary* The proposal pushes lake/log stitching down behind a stable table → scan → plan → read_split contract. Key design points: - *Read model*: unified table → scan → plan → read_split flow; scan is an immutable, cloneable read config; plan() freezes the snapshot boundary and per-split log stop boundary and produces logical splits. - *Three semantics*: standard union read (lake snapshot + log tail as one bounded result), lake-only read (with_lake_only), and PK current-view read (merge lake state with the changelog tail). - *Seam*: the lake/log join point is fixed at Fluss's readableLogEndOffset; the log tail reads from the seam onward, guaranteeing no overlap and no gap so consumers never dedup. - *Split model*: the logical execution unit is uniformly (partition, bucket) for both partitioned and non-partitioned tables; splits are lightweight, serializable, and version-tagged for wire evolution. - *Filter*: an engine-agnostic FlussLakePredicate auto-decomposed into partition / bucket / data predicates, with exact filter semantics (no residual filtering needed upstream). - *PK merge*: current-view semantics (updates take tail value, deletes excluded, untouched keys keep lake value); the merge algorithm (currently hash overlay, O(tail) memory) is an implementation detail, not part of the public contract. Feedback and suggestions are welcome — especially anything you think we've missed. I'm also curious to hear any thoughts on the DataFusion integration. Best regards, Jim
