Hi yuxia, Thanks for the detailed proposal. I have a few additional questions, mainly around production migration and data correctness.
1. Write availability during migration The current proposal requires users to stop native Paimon writers before invoking the procedure, while new writes through Fluss can only start after the Bulk Load, snapshot registration, validation, and datalake enablement have all completed. For a large primary-key table, the Bulk Load may take a long time, so the business write outage could potentially last for hours. Since the final validation compares the table-level Paimon snapshot ID, it also seems that writes to partitions outside the selected Bulk Load scope would advance the snapshot and cause validation to fail. Is an offline, maintenance-window-style migration an intentional limitation of this FIP? For production migration, would it be possible to separate the long-running baseline load from the final write ownership cutover? One possible flow would be: 1. capture a baseline Paimon snapshot; 2. Bulk Load that snapshot while Paimon writers continue running; 3. catch up changes committed after the baseline; 4. stop or fence the Paimon writers briefly; 5. apply the remaining changes, verify consistency, and enable Fluss; 6. switch application writes to Fluss. This would require defining the incremental replay, ordering, deduplication, fencing, rollback, and exact handoff boundary. If online migration is outside the scope of FIP-25, I think it would still be useful to document the expected write outage and the recommended recovery procedure when a long-running migration fails. 2. Enforcement for partially initialized regular partitions For a regular partitioned primary-key table, the proposal allows users to Bulk Load only selected partitions. It states that unloaded partitions do not support Fluss lookups or writes and that users must ensure no writes are issued to them. How is the set of successfully initialized partitions persisted and exposed after the procedure completes? Should Fluss reject lookup and write requests to an unloaded partition, instead of relying only on an operational requirement? Without such enforcement, an accidental write to an unloaded partition may treat the historical KV state as empty and produce incorrect upsert, delete, or changelog behavior. It would also be helpful to clarify whether and how the remaining partitions can be initialized later. 3. Handling of unknown Paimon properties The compatibility section explains that properties unknown to the Paimon version bundled with Fluss are allowed, because they may be application metadata. It also acknowledges that such a property may actually control behavior introduced by a newer Paimon version, in which case promotion could succeed even though the resulting Fluss semantics are different. For a correctness-sensitive operation such as in-place promotion, is silently allowing these properties the right default? Would it be safer to report all unknown properties and require an explicit acknowledgement or allowlist before proceeding? A strict mode that rejects unknown properties by default could also make the compatibility guarantee easier for users to understand. Best regards, Zhe Wang Liting Liu (litiliu) via dev <[email protected]> 于2026年9月9日周三 16:52写道： > Hi Yuxia and Fluss community, > > Thanks for sharing FIP-25. It is a very interesting proposal. While > reading it and comparing it with the current `apache/main` branch, I had a > few questions that may be worth clarifying. > > First, the proposal says that `CALL sys.enable_fluss_on_lake_table(...)` > waits for the `load_lake_data_to_fluss` Bulk Load job, snapshot > registration, and datalake enablement to complete before returning. > > For a large Paimon table, this operation could take a considerable amount > of time. Depending on the SQL Gateway, client, or HTTP/proxy configuration, > the caller might time out while the underlying operation is still running. > In that situation, it may be difficult for the caller to determine the > final outcome. > > Would it make sense for the procedure to return a promotion operation ID > and provide a separate way to query its status? Alternatively, it would be > helpful to document how timeout and cancellation are expected to work. > > Second, I was wondering about failure recovery and retry behavior. > > If Fluss table creation succeeds but the Bulk Load job fails, the proposal > says that the table remains present with datalake disabled. On a subsequent > call, however, the existing table may cause the procedure to skip table > creation and Bulk Load. It would be useful to clarify how the procedure can > distinguish a successfully initialized table from one whose Bulk Load > failed or is still incomplete. > > Could the design consider persisting some information such as the > following to make retry and recovery easier to understand? > > - promotion operation ID; > - source Paimon snapshot ID; > - selected partitions; > - Bulk Load transaction ID. > > It may also be helpful to describe the expected retry, abort, timeout, and > cancellation semantics, and to make sure that a retry cannot enable > datalake before the Bulk Load result has been successfully verified. > > Best regards, > > Liting Liu > > From: yuxia <[email protected]> > Date: Tuesday, September 8, 2026 at 15:23 > To: dev <[email protected]> > Subject: [DISCUSS] FIP-25: Support In-Place Promotion of Existing Paimon > Tables to Datalake-Enabled Fluss Tables > > Hi Fluss community, > > I'd like to start a discussion on FIP-25: Support In-Place Promotion of > Existing Paimon Tables to Datalake-Enabled Fluss Tables[1]. > > This FIP proposes promoting an existing Paimon table to a datalake-enabled > Fluss table in place through: > > CALL sys.enable_fluss_on_lake_table(...); > > "In-place" means reusing the existing Paimon table and historical data > without creating a replacement table or rewriting the data to Paimon. Fluss > creates the corresponding metadata, initializes historical primary-key > state through Bulk Load when necessary, and validates schema and snapshot > consistency before enabling datalake. > > Feedback and suggestions are welcome. > > [1]: > https://cwiki.apache.org/confluence/spaces/FLUSS/pages/406623572/FIP-25+Support+In-Place+Promotion+of+Existing+Paimon+Tables+to+Datalake-Enabled+Fluss+Tables > > Best regards, > Yuxia >
