Hello Jim,
Thank you for the FIP, the general direction looks solid. I have the
following questions:
1. Have we considered timeout, liveness or retry semantics? A hung storage
request or dead server would stall the stream indefinitely without timeout.
This also apply for PK splits where merge must first materialise the whole
changeling tail before emitting anything. It might be worth adding
configurable timeouts. Without this, user will have to resort to wrapping
the kernel with watchdog.
2. On PK hash overlay, the proposal relies on the premise that tail is
bounded and small. There's no defined behaviour when that premise fails.
Could we specify a failure mode? e.g. configurable memory / tail-size cap
surfacing an error such as FlakeLakeError("snapshot too stale") rather than
risking OOMing the host process?
3. Can we include in the proposal on the credential and security story for
object storage? I appreciate that the proposal might be following Java side
implementation, calling it out in the proposal would make it clear.
It's a great FIP, looking forward to your response.
Best regards
Keith
On Thu, Jul 30, 2026 at 4:15 PM Jim Hu <[email protected]> wrote:
> Hi all,
>
> I'd like to start a discussion on FIP-48: Introduce a Union Read Kernel
> for fluss-rust.
>
> The proposal is available here:
>
> https://cwiki.apache.org/confluence/spaces/FLUSS/pages/444334625/FIP-48+Introduce+a+Union+Read+Kernel+for+fluss-rust
>
> Union read is already available to Java engines through the Flink
> connector (FIP-6). FIP-48 brings the same capability to fluss-rust, so that
> non-JVM engines (StarRocks, DataFusion, Python) can read lake-enabled
> tables without embedding a JVM. This FIP was drafted together with beryllw.
>
> In short, it proposes a new fluss-lake crate (Paimon-first) providing:
>
> - A bounded batch read model: one lake snapshot plus its bounded log tail,
> stitched at the readable-snapshot seam with no overlap and no gap
> - A table -> scan -> plan -> read_split contract with distributable,
> serializable, versioned splits; engines schedule splits and consume Arrow
> streams without reimplementing seam or merge logic
> - Three read semantics: append/log tables, primary-key tables (merged
> current view), and lake-only
> - Predicate pushdown with automatic partition/bucket pruning, plan-level
> statistics, and a typed error surface
>
> Looking forward to your feedback and suggestions. Please feel free to
> raise questions in this thread, and I will keep updating FIP-48 based on
> the discussion. If there are no further concerns, I plan to start a vote
> next week.
>
> Best regards,
> Jim
