Hi all, Thank you, Jim, Keith, Leonard, Yang, Junbo for good questions and helping to shape this proposal. Very much appreciated. I've incorporated the feedback, here is what has changed.
Jim and Keith, you're right about the boundary. I'd been experimenting with DataFusion against the client before FIP-48 was proposed, and those sections came from that. When FIP-48 landed I updated the dependency references but not the boundary. FIP-50 consumes the kernel now: new_scan with projection, a FlussLakePredicate, read mode and target parallelism, plan() for splits, read_split for the stream. Split scheduling, predicate translation, residual filtering and re-planning stay with the engine. That leaves one case. A PK table with tiering off has no lake snapshot, so its base image is the KV snapshot from ScanKv. That's a port of KvSnapshotAndLogBatchScanner into the core crate, mirroring Java where it and LakeSnapshotAndLogSplitScanner both sit in fluss-client. The kernel returns NotLakeReadable for these tables anyway. What I'd like to avoid is writing the merge semantics twice. Java has two scanners sharing one SortMergeReader. In Rust the kernel folds a hash overlay and the KV path would fold its own, so merge engines and delete behaviour end up in two places and can drift. Is there a piece worth factoring out so both call it? Leonard, both points are correct, thank you for noticing! Only predicates on primary-key columns may be evaluated below reconciliation. TableDescriptor requires partition keys to be a subset of the primary key and bucket keys a subset of it excluding partition keys, so partition and bucket pruning stay safe. The kernel already does this, pruning works on partitions only and the filter applies to the merged stream, so it binds my non-tiered path, where the crate composes the readers itself. Those predicates report Unsupported, not Inexact, since Inexact still hands the filter to scan. DataUnavailable is statement-level now. It's not only a planning error either: executor.rs maps LogOffsetOutOfRange to it on the read path, so a split can fail after its siblings have emitted. Keith, the naming surface is reserved with only the plain name implemented. fluss.db.t is the current state of the table however it's assembled, so it survives lake-only and changelog arriving later. $lake maps onto with_lake_only, $changelog is reserved for change records. Boundedness stays out of it, since a changelog read is useful as bounded too. Yang, DuckDB would work. DataFusion won it for being a Rust library rather than a database: pluggable catalog and table providers, and it takes the Arrow batches the client already produces without converting at the boundary. The core-client additions are engine-neutral, so a DuckDB extension over the same primitives stays open. One call for FIP-48: column statistics. The plan gives split_count, estimated_total_rows and estimated_total_size, but nothing per column, so filter selectivity stays a guess. -- Anton
