Hi Yuxia, Thanks for the detailed FIP and for all your hard work to push this mammoth effort. I have a few additional questions after reading through the design.
*1. Should the validation also check the lake format?* The creation-time validation checks primary key tables, table.datalake.enabled, and FULL changelog mode, but I did not see a validation for the lake format itself. The design seems to assume Paimon throughout, for example COMPACT snapshots, DELETE records, native deletion vectors, and paimon.deletion-vectors.enabled. Would it make sense to also reject tables whose lake format is not Paimon when table.datalake.deletion-vectors.enabled = true? That way, if support for Iceberg or other lake formats is added later, they can simply be added to the allowlist instead of leaving the current behavior undefined. *2. How is the __rowid preservation contract enforced?* The FIP says external engines may compact or rewrite files as long as they preserve __rowid, but this currently reads as a convention rather than something that is validated. For example: - What happens if an external engine inserts rows without a valid __rowid? - What happens if a rewrite accidentally drops or corrupts __rowid values? Would it be worth adding a validation during the Phase A scan that fails the round if any data file contains null or invalid __rowid values? That would at least turn silent corruption into a visible failure. *3. What happens if compaction never runs?* The readable baseline only advances on COMPACT snapshots. If compaction is disabled, misconfigured, or simply delayed for a long time, it seems that PendingDeletes would continue to grow, LogDv could not be cleaned up, and changelog retention would also be blocked. Should the FIP define any operational guidance here, such as expected compaction frequency, metrics to watch, or backpressure when the system falls too far behind? *4. Is there an escape hatch?* If DV state is ever found to be corrupted, or a production issue is discovered after deployment, is there a way to temporarily fall back to the existing sort-merge based union read path? Having a repair or safety mode could be valuable while the issue is investigated, even if it comes with lower performance. Overall, I think the direction looks very promising. Thanks again for putting together the proposal. Best, Mehul Batra On Wed, Jul 22, 2026 at 4:59 PM yuxia <[email protected]> wrote: > Please disregard my previous email. It was sent by mistake. Sorry for the > confusion. > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "yuxia" <[email protected]> > 收件人: "dev" <[email protected]> > 发送时间: 星期三, 2026年 7 月 22日 下午 6:08:26 > 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary > Key Table Union Read over Paimon > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "yuxia" <[email protected]> > 收件人: "dev" <[email protected]> > 发送时间: 星期三, 2026年 7 月 22日 下午 5:58:30 > 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary > Key Table Union Read over Paimon > > Hi, Liting. > Thanks for raising these concerns. I would like to clarify the exact > expiration scenario in the first point. > > By “expire historical data,” do you mean ordinary Iceberg snapshot > expiration, or an operation that removes live data from the current table > snapshot, such as partition/data expiration? > > For ordinary snapshot expiration, we do not expect orphan RowPosIndex > entries. Snapshot expiration only removes historical snapshots and files > that are no longer referenced by any retained snapshot. Files referenced by > the current DV-readable snapshot must be retained. If files are rewritten > by compaction, the surviving RowIds are scanned from the new files and > their RowPosIndex entries are overwritten with the new positions. > > If Amoro instead commits a new snapshot that logically removes data files > without replacement—for example, by expiring a partition or applying a > data-retention policy—then your concern is valid. Those RowIds would not > appear in any new data file, and no new SST mapping would overwrite their > old RowPosIndex entries. Could you clarify whether this is partition-level > expiration, file-level data expiration, or another Amoro operation? It > would help us define the appropriate support boundary. > > Regarding FIP-27, its goal is to remove mandatory system columns from > regular Fluss lake tables so that all tables do not incur schema pollution > and downstream compatibility impact. DV-enabled tables are an explicit, > opt-in case: the DV mechanism requires internal row identity and bucket > information to rebuild the mapping from RowId to physical file position. > Therefore, while normal lake tables should no longer require these > additional columns, a table with DV explicitly enabled would still carry > the internal columns required by the DV protocol. > > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "Leonard Xu" <[email protected]> > 收件人: "dev" <[email protected]> > 发送时间: 星期三, 2026年 7 月 08日 下午 6:02:29 > 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary > Key Table Union Read over Paimon > > Hi Yuxia. > > Thanks for sharing the FIP. The overall direction looks very promising to > me. Using RowId + RowPosIndex + LakeDv/LogDv to avoid full sort-merge > during primary-key union read makes sense, and the two-phase prepare/switch > protocol also looks like the right direction to reduce the stale-snapshot > window. > > I have several comments that I think are worth clarifying in the FIP > before moving forward: > > RowPosIndex cleanup when files are removed externally > > The design explains how LakeDv entries are cleaned when compaction > replaces files. However, I think we also need to clarify what happens when > files are removed by external operations, such as snapshot expiration, > partition expiration, rewrite, or data lifecycle management. > > RowPosIndex is keyed by RowId and points to a physical file position. If > the underlying Paimon data file is removed outside the normal compaction > path, the RowPosIndex entry may become an orphan entry. Later > updates/deletes may still hit this stale RowPosIndex entry and generate > LakeDv markers for a file that no longer exists. > > Could the FIP clarify whether such external file-removal operations are > forbidden for Fluss-managed Paimon tables, or whether we need an additional > reverse mapping from fileId to RowIds so RowPosIndex can be cleaned by > removed file? > > Compatibility with FIP-27 and system column pollution > > The design introduces a new __rowid column in Paimon files. Since FIP-27 > is trying to remove mandatory system columns from Fluss lake tables, I > think the FIP should clarify how __rowid fits into that direction. > > In particular: > > Is __rowid a hidden/internal column or a normal physical column? > Will external engines see it in SELECT *? > What happens if the user already has a column named __rowid? > How does schema evolution handle this internal column? > Without a clear answer, this may introduce schema pollution or > compatibility issues for downstream engines. > > Merge engine compatibility should be narrowed or clarified > > The FIP says all merge engines are compatible under FULL changelog mode. I > think this may be too optimistic. > > For example, FIRST_ROW has special semantics and does not really support > normal UPDATE/DELETE behavior in Fluss today. But the DV design relies > heavily on -U / -D carrying the old RowId, so the old version can be > located and masked. For merge engines with special update/delete semantics, > it is not obvious that this assumption always holds. > > I suggest narrowing the first version to the default DEDUPLICATE > primary-key table, or adding a detailed correctness explanation and test > plan for each supported merge engine. > > RowPos SST file lifecycle and garbage collection > > The FIP defines remote RowPos SST files under rowPos/{snapshotId}/..., > which are required for TabletServer prepare and recovery. However, the > retention and cleanup policy is not very clear yet. > > If these files are deleted too early, TabletServer recovery may fail > because it cannot re-ingest RowPos SSTs. If they are never deleted, remote > storage may grow indefinitely. > > Could the FIP define the retention rule for RowPos SSTs, and explain how > it is coordinated with: > > DvRocksDB checkpoints, > current/latest DV-readable snapshot, > pending readable rounds, > Paimon snapshot expiration? > LakeDv returned to union read should be filtered by planned files > > During union read, the Paimon scan already knows which data files will be > read. The FIP mentions that LakeDv entries may be filtered by planned file > paths, but I think this should be a required behavior instead of an > optional optimization. > > Otherwise, for delete-heavy tables, returning the whole bucket’s LakeDv > may introduce large network transfer, bitmap deserialization, and memory > overhead. The read cost may simply move from sort-merge to DV transfer. > > I suggest making GetLakeDvSnapshot return only LakeDv entries for the > files planned by the current Paimon scan. > > Overall, I think the FIP is heading in a good direction and the expected > performance improvement is very valuable. The main areas I would like to > see clarified are around lifecycle management, compatibility, and keeping > the read path lightweight. > > Best, > Leonard >
