Hi all,
Thanks Jim and Junbo for driving this FIP. I reviewed the earlier
draft and I think the shape is right.
A few follow-ups from that review, plus some open questions:
Follow-ups from the draft review:
1. §3.10, sort-merge rationale: following the draft discussion, since
both hash overlay and sort-merge buffer the bounded tail, memory looks
equivalent between the two, and the real gains from sort-merge would
be key-ordered output and a fully streaming lake side. Would you
update the switch-point rationale to reflect that?
2. Same section: does the blocker description ("a bucket's files are
bin-packed into key-range-overlapping splits") still hold after
paimon-rust #374, which keeps key-overlapping files within one split?
If #374 is also what guarantees the one-row-per-key current view
across a multi-split bucket, it may be worth stating that dependency
explicitly in the FIP.
3. §3.3, "latest readable snapshot": the DV-tables explanation from
the draft discussion (compaction progress defining readability;
getReadableLakeSnapshot returning the newest fully-readable snapshot
plus per-bucket readableLogEndOffset) was good to clarify, thank you -
would you consider folding it into the FIP text to help other
reviewers?
Open questions:
4. §3.3/§3.5, plan/read consistency: if a worker's read-side
projection or filter drifts from what plan() froze, what happens
today? is there a check, or a silently wrong shape? In the draft
discussion one option raised was splits carrying projection/filter
themselves, which trades against the "Lightweight" constraint. How do
you see this resolving? It looks like the one public-contract decision
that's expensive to change after the vote.
5. §3.5, descriptor_version: if a reader is older than the planner's
descriptor version, is the mismatch detected and rejected with a typed
error, or undefined? What compatibility contract do you have in mind
(reject-newer, reject-older, both)?
6. §3.6, DataUnavailable: is this intended as a distinguishable
"re-plan and retry" signal, as opposed to non-retryable errors?
And on the other boundary - if a split's stop offset can never be
reached, does the read fail after a bound, or can it block
indefinitely?
7. §3.7, exact filter: does the exactness guarantee hold for every
expressible FlussLakePredicate (anything not pushed down is always
applied via arrow-level filtering), or are there predicates where the
SDK falls back and the engine must residual-filter? If it's total,
stating that explicitly would let engines drop their filter operators
with confidence.
8. §3.10, PK tables: which Paimon merge engines are covered by the
current-view semantics aka deduplicate only, or also partial-update /
aggregation / first-row? And for a table configured with an
unsupported engine, what should the caller see - NotLakeReadable at
open, PlanningFailed, or something else?
9. §3.9, append tables - just to confirm the parallelism model: the
unit of parallelism is the (partition, bucket) split and a single
split is one stream, so intra-bucket parallelism of the lake side is
out of scope for this FIP?
Overall +1 on the direction, thank you
-- Anton
чт, 30 июл. 2026 г. в 16:14, Jim Hu <[email protected]>:
>
> Hi all,
>
> I'd like to start a discussion on FIP-48: Introduce a Union Read Kernel for
> fluss-rust.
>
> The proposal is available here:
> https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334625/FIP-48+Introduce+a+Union+Read+Kernel+for+fluss-rust
>
> Union read is already available to Java engines through the Flink connector
> (FIP-6). FIP-48 brings the same capability to fluss-rust, so that non-JVM
> engines (StarRocks, DataFusion, Python) can read lake-enabled tables without
> embedding a JVM. This FIP was drafted together with beryllw.
>
> In short, it proposes a new fluss-lake crate (Paimon-first) providing:
>
> - A bounded batch read model: one lake snapshot plus its bounded log tail,
> stitched at the readable-snapshot seam with no overlap and no gap
> - A table -> scan -> plan -> read_split contract with distributable,
> serializable, versioned splits; engines schedule splits and consume Arrow
> streams without reimplementing seam or merge logic
> - Three read semantics: append/log tables, primary-key tables (merged current
> view), and lake-only
> - Predicate pushdown with automatic partition/bucket pruning, plan-level
> statistics, and a typed error surface
>
> Looking forward to your feedback and suggestions. Please feel free to raise
> questions in this thread, and I will keep updating FIP-48 based on the
> discussion. If there are no further concerns, I plan to start a vote next
> week.
>
> Best regards,
> Jim
