Hi Jim, Thanks for driving FIP-48 and incorporating the community feedback.
Overall, +1 for the updated FIP. I believe the latest revision provides a clear and practical contract for non-JVM engines to perform union reads without reimplementing the lake/log seam and primary-key merge semantics. The remaining limitations, such as memory usage for large PK changelog tails, caller-managed timeouts, and the lack of intra-bucket parallelism, are clearly documented and can be addressed incrementally without changing the core contract. The PoC is also helpful for validating the overall design. I think the proposal is ready to move forward to a vote. Best, Leonard > 2026 8月 13 4:08 下午，Jim Hu <[email protected]> 写道： > > Hi all, > > Thanks everyone for the great feedback and the active discussion so far, > it's really helpful in shaping the proposal. > > Following the design in FIP-48, I've put together a PoC[1] for anyone who > might be interested. > > Given how the discussion has converged, I'd like to start the vote > tomorrow. If anyone would like more time for a final review, just let me > know. > > [1] https://github.com/naivedogger/fluss/tree/feature/fluss-union-read > > Best regards, > Jim Hu > > Jim Hu <[email protected]> 于2026年8月7日周五 15:53写道： > >> Hi Keith, >> >> Thanks for the review and the kind words. Good points! >> >>> 1. Have we considered timeout, liveness or retry semantics? ... It might >> be worth adding configurable timeouts. Without this, user will have to >> resort to wrapping the kernel with watchdog. >> >> The concern is valid, but I think it’s better not to add a reader-level >> timeout, to stay aligned with fluss-java. In fluss-java, >> LogScanner::poll(Duration) and BatchScanner::pollBatch(Duration) take a >> per-call timeout and return empty on timeout; higher-level retry and >> cancellation are left to the caller. StarRocks and Flink already manage >> query-level timeouts around connector reads, so a reader-level timeout >> would just duplicate that. Transient RPC errors are retried by the >> underlying client; a stuck or dead server shows up as ConnectionError. For >> slow PK tail reads, the caller can wrap stream.next() with their own >> timeout. >> >>> 2. On PK hash overlay, the proposal relies on the premise that tail is >> bounded and small. There's no defined behaviour when that premise fails. >> Could we specify a failure mode? >> >> Good point, the OOM risk is real. But I think it’s better to leave this >> out of V1 for now and track it as future work. I have noted this in the >> FIP. What do you think? >> >>> 3. Can we include in the proposal on the credential and security story >> for object storage? ... calling it out in the proposal would make it clear. >> >> Added a "Security and credentials" section to the FIP. In short: >> fluss-lake doesn't manage credentials itself. Auth for object storage (S3, >> OSS, GCS, etc.) is delegated to the underlying storage client. Credentials >> are resolved at reader construction time, never logged, and never put into >> splits. Splits only carry storage-relative paths. >> >> All changes are on the Confluence page. Appreciate the review! >> >> Best regards, >> Jim >> >> >> 2026年7月31日 下午6:58，Keith Lee <[email protected]> 写道： >> >> Hello Jim, >> >> Thank you for the FIP, the general direction looks solid. I have the >> following questions: >> >> 1. Have we considered timeout, liveness or retry semantics? A hung storage >> request or dead server would stall the stream indefinitely without timeout. >> This also apply for PK splits where merge must first materialise the whole >> changeling tail before emitting anything. It might be worth adding >> configurable timeouts. Without this, user will have to resort to wrapping >> the kernel with watchdog. >> 2. On PK hash overlay, the proposal relies on the premise that tail is >> bounded and small. There's no defined behaviour when that premise fails. >> Could we specify a failure mode? e.g. configurable memory / tail-size cap >> surfacing an error such as FlakeLakeError("snapshot too stale") rather than >> risking OOMing the host process? >> 3. Can we include in the proposal on the credential and security story for >> object storage? I appreciate that the proposal might be following Java side >> implementation, calling it out in the proposal would make it clear. >> >> It's a great FIP, looking forward to your response. >> >> Best regards >> Keith >> >> >>
