Hi all, Thanks everyone for the great feedback and the active discussion so far, it's really helpful in shaping the proposal.
Following the design in FIP-48, I've put together a PoC[1] for anyone who might be interested. Given how the discussion has converged, I'd like to start the vote tomorrow. If anyone would like more time for a final review, just let me know. [1] https://github.com/naivedogger/fluss/tree/feature/fluss-union-read Best regards, Jim Hu Jim Hu <[email protected]> 于2026年8月7日周五 15:53写道： > Hi Keith, > > Thanks for the review and the kind words. Good points! > > > 1. Have we considered timeout, liveness or retry semantics? ... It might > be worth adding configurable timeouts. Without this, user will have to > resort to wrapping the kernel with watchdog. > > The concern is valid, but I think it’s better not to add a reader-level > timeout, to stay aligned with fluss-java. In fluss-java, > LogScanner::poll(Duration) and BatchScanner::pollBatch(Duration) take a > per-call timeout and return empty on timeout; higher-level retry and > cancellation are left to the caller. StarRocks and Flink already manage > query-level timeouts around connector reads, so a reader-level timeout > would just duplicate that. Transient RPC errors are retried by the > underlying client; a stuck or dead server shows up as ConnectionError. For > slow PK tail reads, the caller can wrap stream.next() with their own > timeout. > > > 2. On PK hash overlay, the proposal relies on the premise that tail is > bounded and small. There's no defined behaviour when that premise fails. > Could we specify a failure mode? > > Good point, the OOM risk is real. But I think it’s better to leave this > out of V1 for now and track it as future work. I have noted this in the > FIP. What do you think? > > > 3. Can we include in the proposal on the credential and security story > for object storage? ... calling it out in the proposal would make it clear. > > Added a "Security and credentials" section to the FIP. In short: > fluss-lake doesn't manage credentials itself. Auth for object storage (S3, > OSS, GCS, etc.) is delegated to the underlying storage client. Credentials > are resolved at reader construction time, never logged, and never put into > splits. Splits only carry storage-relative paths. > > All changes are on the Confluence page. Appreciate the review! > > Best regards, > Jim > > > 2026年7月31日 下午6:58，Keith Lee <[email protected]> 写道： > > Hello Jim, > > Thank you for the FIP, the general direction looks solid. I have the > following questions: > > 1. Have we considered timeout, liveness or retry semantics? A hung storage > request or dead server would stall the stream indefinitely without timeout. > This also apply for PK splits where merge must first materialise the whole > changeling tail before emitting anything. It might be worth adding > configurable timeouts. Without this, user will have to resort to wrapping > the kernel with watchdog. > 2. On PK hash overlay, the proposal relies on the premise that tail is > bounded and small. There's no defined behaviour when that premise fails. > Could we specify a failure mode? e.g. configurable memory / tail-size cap > surfacing an error such as FlakeLakeError("snapshot too stale") rather than > risking OOMing the host process? > 3. Can we include in the proposal on the credential and security story for > object storage? I appreciate that the proposal might be following Java side > implementation, calling it out in the proposal would make it clear. > > It's a great FIP, looking forward to your response. > > Best regards > Keith > > >
