shangxinli commented on issue #18135:
URL: https://github.com/apache/hudi/issues/18135#issuecomment-5743405449
Started implementing this and hit something that changes the design, so
flagging it before going further.
**The `FILES` metadata approach cannot address files outside the table base
path.**
I built the write path as discussed above — REGISTER_ONLY partitions
registered into the metadata table's `FILES` partition using the existing
external-file convention (`ExternalFilePathUtil`, the `_<commitTime>_hudiext`
marker from HUDI-6648 / #9367), with the file group id assigned once at
registration. That part works: the stats land in `FILES` and the marker
round-trips.
It falls over at read time. A three-tier end-to-end test (hot `FULL_RECORD`
/ warm `METADATA_ONLY` / cold `REGISTER_ONLY`) fails on the cold tier with:
```
SparkFileNotFoundException:
file:/.../target/datestr=2024-04-02/part-00002-....parquet does not exist
```
The file exists, but under the *bootstrap source* path, not the target table
path. The cause is structural rather than a wiring bug —
`HoodieMetadataPayload.getFileList` builds each entry as:
```java
new StoragePath(partitionPath, e.getKey()) //
HoodieMetadataPayload.java:508
```
where `partitionPath` is derived from the table's own base path. A `FILES`
entry is a *file name within a partition of this table*; there is nowhere to
record "and this file actually lives elsewhere". The external-file mechanism
was built for files already sitting under the table — e.g. #19869, another
engine writing into the table's own directory — which is a different situation
from bootstrap, whose whole premise is that the data is somewhere else.
Mapping a Hudi file to an external source path is exactly what
`BootstrapFileMapping` and the bootstrap index exist for. But reusing the index
here does not work either: `HoodieFileGroupReader` stitches skeleton and data
positionally (`makeBootstrapBaseFileIterator`), and REGISTER_ONLY has no
skeleton. Synthesizing one needs the row count, which needs reading the file,
which is the cost the mode exists to avoid.
**Proposed narrowing:** scope REGISTER_ONLY to *in-place* registration,
where the table base path is the existing data location. Cold files are then
already under the table, `FILES` addresses them correctly, and no read-path
change is needed at all — Spark already serves the Hudi metadata columns as
null for a file that lacks them, because they are nullable
(`HoodieSchemaUtils.java:105`) and `HoodieParquetReadSupport` extends stock
`ParquetReadSupport`, so `clipParquetSchema` synthesizes the missing optional
column. For the cold-storage use case this is arguably the better shape anyway:
nothing is copied and the archived files are never touched.
Two questions for @vinothchandar / @nsivabalan / @vamshipasunuru1:
1. Is in-place registration the intended scope, or do you want `FILES`
extended to carry a source path so a separate bootstrap target keeps working?
The latter is a storage-format change, which felt bigger than what was sketched
above.
2. If in-place: `METADATA_ONLY` is the awkward tier, since its skeleton
files would land in the same partition directories as the source data. Should
in-place support only `FULL_RECORD` + `REGISTER_ONLY`?
Separately, one sharp edge worth recording regardless of which way this
goes: `HoodieFileGroupReaderBasedFileFormat` recovers the file group via
`FSUtils.getFileId`, which is `split("_", 2)[0]`. An external file whose name
contains an underscore (`part_0001.parquet`, or Hive's `000000_0`) will not
match its file slice and silently falls through to the plain base-file branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
