Please disregard my previous email. It was sent by mistake. Sorry for the confusion.
Best regards, Yuxia ----- 原始邮件 ----- 发件人: "yuxia" <[email protected]> 收件人: "dev" <[email protected]> 发送时间: 星期三, 2026年 7 月 22日 下午 6:08:26 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon Best regards, Yuxia ----- 原始邮件 ----- 发件人: "yuxia" <[email protected]> 收件人: "dev" <[email protected]> 发送时间: 星期三, 2026年 7 月 22日 下午 5:58:30 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon Hi, Liting. Thanks for raising these concerns. I would like to clarify the exact expiration scenario in the first point. By “expire historical data,” do you mean ordinary Iceberg snapshot expiration, or an operation that removes live data from the current table snapshot, such as partition/data expiration? For ordinary snapshot expiration, we do not expect orphan RowPosIndex entries. Snapshot expiration only removes historical snapshots and files that are no longer referenced by any retained snapshot. Files referenced by the current DV-readable snapshot must be retained. If files are rewritten by compaction, the surviving RowIds are scanned from the new files and their RowPosIndex entries are overwritten with the new positions. If Amoro instead commits a new snapshot that logically removes data files without replacement—for example, by expiring a partition or applying a data-retention policy—then your concern is valid. Those RowIds would not appear in any new data file, and no new SST mapping would overwrite their old RowPosIndex entries. Could you clarify whether this is partition-level expiration, file-level data expiration, or another Amoro operation? It would help us define the appropriate support boundary. Regarding FIP-27, its goal is to remove mandatory system columns from regular Fluss lake tables so that all tables do not incur schema pollution and downstream compatibility impact. DV-enabled tables are an explicit, opt-in case: the DV mechanism requires internal row identity and bucket information to rebuild the mapping from RowId to physical file position. Therefore, while normal lake tables should no longer require these additional columns, a table with DV explicitly enabled would still carry the internal columns required by the DV protocol. Best regards, Yuxia ----- 原始邮件 ----- 发件人: "Leonard Xu" <[email protected]> 收件人: "dev" <[email protected]> 发送时间: 星期三, 2026年 7 月 08日 下午 6:02:29 主题: Re: [DISCUSS] FIP-47: Introduce Deletion Vectors to accelerate Primary Key Table Union Read over Paimon Hi Yuxia. Thanks for sharing the FIP. The overall direction looks very promising to me. Using RowId + RowPosIndex + LakeDv/LogDv to avoid full sort-merge during primary-key union read makes sense, and the two-phase prepare/switch protocol also looks like the right direction to reduce the stale-snapshot window. I have several comments that I think are worth clarifying in the FIP before moving forward: RowPosIndex cleanup when files are removed externally The design explains how LakeDv entries are cleaned when compaction replaces files. However, I think we also need to clarify what happens when files are removed by external operations, such as snapshot expiration, partition expiration, rewrite, or data lifecycle management. RowPosIndex is keyed by RowId and points to a physical file position. If the underlying Paimon data file is removed outside the normal compaction path, the RowPosIndex entry may become an orphan entry. Later updates/deletes may still hit this stale RowPosIndex entry and generate LakeDv markers for a file that no longer exists. Could the FIP clarify whether such external file-removal operations are forbidden for Fluss-managed Paimon tables, or whether we need an additional reverse mapping from fileId to RowIds so RowPosIndex can be cleaned by removed file? Compatibility with FIP-27 and system column pollution The design introduces a new __rowid column in Paimon files. Since FIP-27 is trying to remove mandatory system columns from Fluss lake tables, I think the FIP should clarify how __rowid fits into that direction. In particular: Is __rowid a hidden/internal column or a normal physical column? Will external engines see it in SELECT *? What happens if the user already has a column named __rowid? How does schema evolution handle this internal column? Without a clear answer, this may introduce schema pollution or compatibility issues for downstream engines. Merge engine compatibility should be narrowed or clarified The FIP says all merge engines are compatible under FULL changelog mode. I think this may be too optimistic. For example, FIRST_ROW has special semantics and does not really support normal UPDATE/DELETE behavior in Fluss today. But the DV design relies heavily on -U / -D carrying the old RowId, so the old version can be located and masked. For merge engines with special update/delete semantics, it is not obvious that this assumption always holds. I suggest narrowing the first version to the default DEDUPLICATE primary-key table, or adding a detailed correctness explanation and test plan for each supported merge engine. RowPos SST file lifecycle and garbage collection The FIP defines remote RowPos SST files under rowPos/{snapshotId}/..., which are required for TabletServer prepare and recovery. However, the retention and cleanup policy is not very clear yet. If these files are deleted too early, TabletServer recovery may fail because it cannot re-ingest RowPos SSTs. If they are never deleted, remote storage may grow indefinitely. Could the FIP define the retention rule for RowPos SSTs, and explain how it is coordinated with: DvRocksDB checkpoints, current/latest DV-readable snapshot, pending readable rounds, Paimon snapshot expiration? LakeDv returned to union read should be filtered by planned files During union read, the Paimon scan already knows which data files will be read. The FIP mentions that LakeDv entries may be filtered by planned file paths, but I think this should be a required behavior instead of an optional optimization. Otherwise, for delete-heavy tables, returning the whole bucket’s LakeDv may introduce large network transfer, bitmap deserialization, and memory overhead. The read cost may simply move from sort-merge to DV transfer. I suggest making GetLakeDvSnapshot return only LakeDv entries for the files planned by the current Paimon scan. Overall, I think the FIP is heading in a good direction and the expected performance improvement is very valuable. The main areas I would like to see clarified are around lifecycle management, compatibility, and keeping the read path lightweight. Best, Leonard
