Hi Anton, Thanks for the careful review, all good points. I've updated the FIP, responses below.
> §3.10, sort-merge rationale: ... Would you update the switch-point rationale > to reflect that? Done. The section now says both approaches buffer the bounded tail so memory is comparable, and the real benefit of sort-merge would be key-ordered output plus a streaming lake side. The switch stays internal and only kicks in once the lake reader can give us bucket-wide primary-key ordering. > Same section: does the blocker description still hold after paimon-rust #374 > ... it may be worth stating that dependency explicitly. Yes, #374 is what guarantees one-row-per-key across splits, and I've added that as an explicit assumption in the "V1 strategy: hash overlay" section. > §3.3, "latest readable snapshot": ... would you consider folding it into the > FIP text? Yes, added. The planner requests the readable snapshot from the server. For DV-enabled Paimon tables the readable snapshot can lag the latest committed one until compaction catches up; for non-DV formats they're the same. If there's no readable snapshot yet, the planner falls back to a full log read. > §3.3/§3.5, plan/read consistency: if a worker's read-side projection or > filter drifts ... How do you see this resolving? This follows the paimon-rust model: splits stay lightweight and carry only positioning, while projection/filter live on the reader. The upstream engine must reconstruct the same FlussLakeScan configuration on every worker. A mismatch is a caller-side programming error and can produce wrong-shaped output. We considered embedding a config fingerprint in splits but rejected it to keep splits small; it's documented under Rejected Alternatives. > §3.5, descriptor_version: ... compatibility contract? Reject-newer, accept-older. A reader that receives a split with a higher descriptor_version than it supports fails with FlussLakeError::IncompatibleSplitVersion and reports both versions. Older supported versions are decoded through internal dispatch. > §3.6, DataUnavailable: ... can it block indefinitely? DataUnavailable is the "plan is stale, re-plan and retry" signal. We intentionally don't add a reader-level no-progress timeout; callers wrap stream.next() with their own timeout/cancellation, matching fluss-java's poll(Duration) behavior. A stop offset that becomes unreachable due to truncation surfaces as DataUnavailable, not an indefinite block. > §3.7, exact filter: ... If it's total, stating that explicitly would let > engines drop their filter operators with confidence. Yes, the guarantee is total for every expressible FlussLakePredicate passed to with_filter(). Engines can safely drop residual filters for those predicates; only unsupported expressions stay as engine-side residuals. > §3.10, PK tables: which Paimon merge engines are covered ... what should the > caller see? V1 only supports deduplicate for PK union read. Other engines fail at plan() time with FlussLakeError::UnsupportedMergeEngine. We don't reject at open() because lake-only reads on the same table are still valid; NotLakeReadable is reserved for non-lake-readable tables. > §3.9, append tables - just to confirm the parallelism model ... intra-bucket > parallelism of the lake side is out of scope? The unit of parallelism is (partition, bucket), one stream per split, and intra-bucket file-level parallelism is out of scope for V1. Ordering across buckets is not guaranteed. All changes are on the Confluence page. Thanks again for the detailed feedback. Best Regards, Jim > 2026年8月6日 上午11:42，Jim Hu <[email protected]> 写道： > > > Hi Anton and Keith, > > Thank you very much for the detailed and thoughtful feedback. These are all > valid points that deserve clearer treatment in the FIP, particularly around > the plan/read consistency contract, descriptor compatibility, retry and > liveness semantics, resource bounds, PK merge semantics, split parallelism, > and object-storage security. > > I’m considering these points and revising the FIP accordingly. I’ll follow up > with a revised version. > > Best, > Jim > >> 2026年8月6日 上午10:47，Jim Hu <[email protected]> 写道： >> >> Hi, all >> >> Thanks Anton and Keith for >> >> Keith Lee <[email protected] <mailto:[email protected]>> 于2026年7月31日周五 >> 18:59写道： >>> Hello Jim, >>> >>> Thank you for the FIP, the general direction looks solid. I have the >>> following questions: >>> >>> 1. Have we considered timeout, liveness or retry semantics? A hung storage >>> request or dead server would stall the stream indefinitely without timeout. >>> This also apply for PK splits where merge must first materialise the whole >>> changeling tail before emitting anything. It might be worth adding >>> configurable timeouts. Without this, user will have to resort to wrapping >>> the kernel with watchdog. >>> 2. On PK hash overlay, the proposal relies on the premise that tail is >>> bounded and small. There's no defined behaviour when that premise fails. >>> Could we specify a failure mode? e.g. configurable memory / tail-size cap >>> surfacing an error such as FlakeLakeError("snapshot too stale") rather than >>> risking OOMing the host process? >>> 3. Can we include in the proposal on the credential and security story for >>> object storage? I appreciate that the proposal might be following Java side >>> implementation, calling it out in the proposal would make it clear. >>> >>> It's a great FIP, looking forward to your response. >>> >>> Best regards >>> Keith >>> >>> On Thu, Jul 30, 2026 at 4:15 PM Jim Hu <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> > Hi all, >>> > >>> > I'd like to start a discussion on FIP-48: Introduce a Union Read Kernel >>> > for fluss-rust. >>> > >>> > The proposal is available here: >>> > >>> > https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334625/FIP-48+Introduce+a+Union+Read+Kernel+for+fluss-rust >>> > >>> > Union read is already available to Java engines through the Flink >>> > connector (FIP-6). FIP-48 brings the same capability to fluss-rust, so >>> > that >>> > non-JVM engines (StarRocks, DataFusion, Python) can read lake-enabled >>> > tables without embedding a JVM. This FIP was drafted together with >>> > beryllw. >>> > >>> > In short, it proposes a new fluss-lake crate (Paimon-first) providing: >>> > >>> > - A bounded batch read model: one lake snapshot plus its bounded log tail, >>> > stitched at the readable-snapshot seam with no overlap and no gap >>> > - A table -> scan -> plan -> read_split contract with distributable, >>> > serializable, versioned splits; engines schedule splits and consume Arrow >>> > streams without reimplementing seam or merge logic >>> > - Three read semantics: append/log tables, primary-key tables (merged >>> > current view), and lake-only >>> > - Predicate pushdown with automatic partition/bucket pruning, plan-level >>> > statistics, and a typed error surface >>> > >>> > Looking forward to your feedback and suggestions. Please feel free to >>> > raise questions in this thread, and I will keep updating FIP-48 based on >>> > the discussion. If there are no further concerns, I plan to start a vote >>> > next week. >>> > >>> > Best regards, >>> > Jim >
