shangxinli commented on issue #18135: URL: https://github.com/apache/hudi/issues/18135#issuecomment-5743428837
Correcting my own suggestion above: **in-place registration is not achievable through the bootstrap operation**, so option 1 in my previous comment is off the table. `HoodieSparkSqlWriter` hard-asserts the two paths differ: ```scala assert(!path.equals(bootstrapBasePath), "Bootstrap base path and Hudi table base path must be different") // HoodieSparkSqlWriter.scala:732 ``` I tried it and every three-tier test fails on that assertion. So the two constraints are contradictory: - bootstrap requires source path ≠ table base path, and - a `FILES` entry can only name a file under the table base path (`HoodieMetadataPayload.getFileList`, `new StoragePath(partitionPath, fileName)`). Which means REGISTER_ONLY **cannot be a bootstrap mode backed by `FILES` metadata**. Not a wiring problem — the two requirements cannot both hold. That leaves, as far as I can see, three directions: 1. **Extend the metadata table to carry a source path** for registered files, so a `FILES` entry can address a file outside the table. Storage-format change, but it is the only option that keeps REGISTER_ONLY inside `bootstrap` with a separate target. 2. **Make this not a bootstrap mode at all** — a separate "register existing partitions" operation against a table created over the existing data, which is the shape #19869 already uses (replace commits with the `_hudiext` marker). This sidesteps the assertion entirely, but it is a different feature from what this issue describes, and it does not compose with `FULL_RECORD` / `METADATA_ONLY` tiering in one pass. 3. **Give the read path a third case** — a base file with an external data file and no skeleton, read directly with the metadata columns served as null. The blocker here is discovery rather than reading: with no skeleton there is no file under the table for `FILES` to list, so this still needs (1) underneath. My read is that (1) is the only one that preserves the feature as specified, and it is a bigger change than what was sketched earlier in this thread. Before going further I would rather hear which direction you would accept — @vinothchandar @nsivabalan @vamshipasunuru1. For what it is worth, the pieces that are agnostic to this decision are already working: the `REGISTER_ONLY` enum value, a `DateBasedBootstrapModeSelector` that tiers partitions by date, and the write-stat construction that records files from a listing alone without opening them. Those survive any of the three directions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
