shangxinli commented on issue #18135: URL: https://github.com/apache/hudi/issues/18135#issuecomment-5744013432
Following up on the three directions above: I went with **(1) extend the metadata table to carry a source path**, and it works end to end. Branch: https://github.com/shangxinli/hudi/tree/feat/register-only-bootstrap The core change is one optional field. `HoodieMetadataFileInfo` gains `sourceBasePath` (nullable, default null), and `HoodieMetadataPayload.getFileList` resolves an entry against it instead of against the partition path when it is set. A null field gives exactly today's behavior, and older records deserialize unchanged. On the write side `HoodieWriteStat` carries the same value through `FilesIndexer` into the FILES partition. Two more places assumed every file lives under the table base path, and both only showed up at runtime: 1. `AbstractTableFileSystemView.addFilesToView(List)` and `BaseHoodieTableFileIndex.generatePartitionFileSlicesPostROTablePathFilter` recovered each file's partition from the file's own parent directory, which threw `does not belong to base-path`. The listing now stays keyed by relative partition path from the point it is produced, so the partition is never re-derived. This seems worth doing on its own merits. 2. `SparkHoodieTableFileIndex.scala:444` treats a partition as absent if its directory does not exist under the base path. A registered partition has no directory of its own, so any predicate naming one pruned to zero rows while an unfiltered scan returned them. Gated on the new table property. A three-tier functional test (`FULL_RECORD` / `METADATA_ONLY` / `REGISTER_ONLY` over one source table) passes: `SELECT *` returns all tiers, a predicate on the cold partition returns its rows, meta columns are null only for the registered tier, and nothing is copied into the table. `TestBootstrapRead` still passes unchanged; I have not run full CI yet. @vinothchandar — the storage-format piece is the one I would most like a read on before going further: an optional `sourceBasePath` on `HoodieMetadataFileInfo` is the smallest change I found that lets a FILES entry address a file the table does not own. If you would rather this be expressed some other way, better to know now than after phases 2-4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
