Hi all, I'd like to start a discussion on FIP-48: Introduce a Union Read Kernel for fluss-rust.
The proposal is available here: https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334625/FIP-48+Introduce+a+Union+Read+Kernel+for+fluss-rust Union read is already available to Java engines through the Flink connector (FIP-6). FIP-48 brings the same capability to fluss-rust, so that non-JVM engines (StarRocks, DataFusion, Python) can read lake-enabled tables without embedding a JVM. This FIP was drafted together with beryllw. In short, it proposes a new fluss-lake crate (Paimon-first) providing: - A bounded batch read model: one lake snapshot plus its bounded log tail, stitched at the readable-snapshot seam with no overlap and no gap - A table -> scan -> plan -> read_split contract with distributable, serializable, versioned splits; engines schedule splits and consume Arrow streams without reimplementing seam or merge logic - Three read semantics: append/log tables, primary-key tables (merged current view), and lake-only - Predicate pushdown with automatic partition/bucket pruning, plan-level statistics, and a typed error surface Looking forward to your feedback and suggestions. Please feel free to raise questions in this thread, and I will keep updating FIP-48 based on the discussion. If there are no further concerns, I plan to start a vote next week. Best regards, Jim
