Hi Anton, Thanks for incorporating the feedback. The revised FIP resolves both of my concerns: PK predicates are now explicitly constrained around reconciliation, and DataUnavailable is clearly defined as a statement-level failure with full-query re-execution.
I have only one small wording clarification. The pushdown section first says that mutable-column predicates on PK tables report Unsupported, but later says predicates on the “log and KV paths” become Inexact after the predicate port lands. For the PK KV-snapshot-plus-tail path, Inexact could still pass the predicate into scan. Could we clarify that the latter applies to log tables only, unless the KV scanner evaluates the predicate after reconciliation? Regarding non-tiered PK tables, perhaps we could build on the foundation established by FIP-48 and consider supporting the KV snapshot as another base-image source in a follow-up. This could bring tiered and non-tiered PK reads under the same engine-neutral current-view contract without blocking FIP-50. @Jim, since this may affect the future scope and boundary of FIP-48, I would also be interested in your thoughts. Otherwise, the updated boundary and semantics look good to me. Thanks again for the detailed update! Best, Leonard > 2026 8月 14 8:14 下午，Anton Borisov <[email protected]> 写道： > > Hi all, > > Thank you, Jim, Keith, Leonard, Yang, Junbo for good questions and > helping to shape this proposal. Very much appreciated. > I've incorporated the feedback, here is what has changed. > > Jim and Keith, you're right about the boundary. I'd been experimenting > with DataFusion against the client before FIP-48 was proposed, and > those sections came from that. When FIP-48 landed I updated the > dependency references but not the boundary. > > FIP-50 consumes the kernel now: new_scan with projection, a > FlussLakePredicate, read mode and target parallelism, plan() for > splits, read_split for the stream. Split scheduling, predicate > translation, residual filtering and re-planning stay with the engine. > > That leaves one case. A PK table with tiering off has no lake > snapshot, so its base image is the KV snapshot from ScanKv. That's a > port of KvSnapshotAndLogBatchScanner into the core crate, mirroring > Java where it and LakeSnapshotAndLogSplitScanner both sit in > fluss-client. The kernel returns NotLakeReadable for these tables > anyway. > > What I'd like to avoid is writing the merge semantics twice. Java has > two scanners sharing one SortMergeReader. In Rust the kernel folds a > hash overlay and the KV path would fold its own, so merge engines and > delete behaviour end up in two places and can drift. Is there a piece > worth factoring out so both call it? > > Leonard, both points are correct, thank you for noticing! > > Only predicates on primary-key columns may be evaluated below > reconciliation. TableDescriptor requires partition keys to be a subset > of the primary key and bucket keys a subset of it excluding partition > keys, so partition and bucket pruning stay safe. The kernel already > does this, pruning works on partitions only and the filter applies to > the merged stream, so it binds my non-tiered path, where the crate > composes the readers itself. > Those predicates report Unsupported, not Inexact, since Inexact still > hands the filter to scan. > > DataUnavailable is statement-level now. It's not only a planning error > either: executor.rs maps LogOffsetOutOfRange to it on the read path, > so a split can fail after its siblings have emitted. > > Keith, the naming surface is reserved with only the plain name > implemented. fluss.db.t is the current state of the table however it's > assembled, so it survives lake-only and changelog arriving later. > $lake maps onto with_lake_only, $changelog is reserved for change > records. Boundedness stays out of it, since a changelog read is useful > as bounded too. > > Yang, DuckDB would work. DataFusion won it for being a Rust library > rather than a database: pluggable catalog and table providers, and it > takes the Arrow batches the client already produces without converting > at the boundary. The core-client additions are engine-neutral, so a > DuckDB extension over the same primitives stays open. > > One call for FIP-48: column statistics. The plan gives split_count, > estimated_total_rows and estimated_total_size, but nothing per column, > so filter selectivity stays a guess. > > -- Anton
