Hi Anton, Thanks for driving this FIP. I think adding native DataFusion query capabilities to the Rust client is very valuable, and I went through the proposal.
Jim and Keith have already covered the FIP-48/FIP-50 ownership boundary and explicit read-mode semantics well. I have two potential correctness comments: 1. For PK union reads, could non-key predicates remain above reconciliation? For example, suppose the lake contains `(id=1, status='open')`, while the log tail updates it to `status='closed'`. For a query filtering on `status='open'`, pushing the predicate independently into both inputs would remove the log update but retain the stale lake row. Reconciliation would then incorrectly return that row. Could pre-reconciliation pushdown therefore be limited to safe pruning, such as partition, bucket, and primary-key predicates, while other filters remain above `FlussPkDedupExec`? 2. Could we clarify the statement-level handling of `DataUnavailable`? FIP-48 requires discarding the old plan’s output and creating a new plan. If one DataFusion partition fails after others have already emitted batches, retrying only that partition with a new plan could mix different snapshot/log boundaries. Would it make sense to specify that `DataUnavailable` fails the entire statement, cancels the remaining partitions, and requires all previously emitted output to be discarded before replanning the full query? Best, Leonard > 2026 8月 14 3:25 下午，Keith Lee <[email protected]> 写道： > > Hello Anton, > > Thank you for the FIP. It's a great proposal as it extensively thought out > the implementation (crate boundaries, responsibilities, implementation > challenges) and trade offs, with following questions and comments. > > 1. I echo Jim's question on implementation boundary with FIP-48. If there's > necessity that drove the overlap, it's worth calling out and discuss > potential changes to either FIP to reduce code maintenance. > 2. Currently the unqualified table name implicitly means bounded union read > as a side effect of planning, but I think this deserves to be an explicit > semantic decision since it is a one-way door. When we later add lake-only, > changelog or unbounded reads, the query text itself cannot express the > distinction (boundedness in DataFusion is a source property, not a query > property), so the choice has to live somewhere deliberate like a table name > convention, a registration option or session config. Should we reserve that > surface (or at very least for bounded vs unbounded) now even if > implementation is deferred, e.g. plain name = bounded union read, $lake = > lake-only, $changelog = the eventual unbounded surface, and snapshot reads > via the pinned catalog provider? > > Again great proposal and overall +1 from me once the above are addressed. > > Best regards > Keith > > On Fri, 14 Aug 2026 at 04:57, Jim Hu <[email protected]> wrote: > >>> >>> Hi Anton, >>> >>> Thanks for putting together this detailed FIP. The motivation for an >>> embeddable query engine on top of the Rust client makes sense to me, and >> I >>> also like that DataFusion is kept outside the core client. The discussion >>> around correctness and resource constraints is very helpful. >>> >>> I may have missed some details, but I have a question about how this >>> proposal is intended to work with FIP-48. >>> >>> I see that tiered reads are listed as depending on FIP-48, and that lake >>> data >>> enters DataFusion through the lake reader. What I could not tell is >> whether >>> fluss-datafusion consumes the complete Union Read plans, opaque splits >> and >>> readers defined by FIP-48, or only uses the lake-format reader while >> owning >>> the boundary handling and reconciliation itself. >>> >>> My understanding of FIP-48 is that snapshot/log boundary freezing, >> lake/log >>> stitching and PK current-view reconciliation belong to the engine-neutral >>> fluss-lake layer. If FIP-50 intends to place some of these >> responsibilities >>> in DataFusion instead, should the boundary defined by FIP-48 be adjusted? >>> Otherwise, could we end up with two implementations of seam handling, >> retry >>> semantics and PK reconciliation? >>> >>> Thanks again for this great FIP! >>> >>> Best, >>> Jim >>> >>
