yihua opened a new pull request, #20259: URL: https://github.com/apache/hudi/pull/20259
### Describe the issue this Pull Request addresses closes #20258 part of #20257 On table version 10 every read of a metadata table native HFile log file opened it 3 times, made 2 file status calls (4 when the size is unknown) and downloaded the whole file 3 times (footer read, schema and meta info load, record iterator), although the file slice listing already carries the length. #11935 (1.0.0) removed the shared HFile reader, so each read creates its own; #13567 and #13946 (1.1.0) made each new reader download the whole file (up to 50 MB); and the unreleased native log path (#19072, #19129, #19118) added the footer read and the `lookupRecords` call without the known size. ### Summary and Changelog A metadata table HFile log file is now downloaded once per reader and its length is not looked up again when the listing already has it. - `HFileReaderFactory` keeps the downloaded content of a file under `hoodie.metadata.file.cache.max.size.mb` and serves every `HFileReader` it creates from it; `HoodieNativeAvroHFileReader.close()` drops it. - New `FSUtils.getNativeLogFilePathInfo` returns the listing path info of a native log file, or one built from its size (fetched once and kept on the `HoodieLogFile` when unknown). The native log file reader, data and delete blocks and the LSM log iterator pass it to the footer reads (new `StoragePathInfo` overloads of `FileFormatUtils.readFooter` and `NativeLogFooterMetadata.readSchemaFromNativeLogFile`), `getFileRecordIterator` and `lookupRecords`. - `HoodieReaderContext.lookupRecords` takes a `StoragePathInfo` (unreleased API from #19118). `HoodieFileReaderFactory.getFileReader(StoragePathInfo, ...)` supports `VORTEX` like the `StoragePath` variant. - Tests: `TestHoodieFileGroupReaderNativeLogs#testNativeHFileLogFileReadsPerFile` (fails on master with `expected: <2> but was: <3>` opens per file) and `TestHoodieHFileReaderWriter#testReaderDownloadsFileOnce` (fails on master with `expected: <1> but was: <5>`), plus cases in `TestHFileReaderFactory` and `TestHoodieReaderContext`. ### Impact Fewer storage requests on every metadata table read path of table version 10. In a local Spark harness (COW, SIMPLE index, column stats + partition stats, 9 writes + 12 reads), calls on metadata table data files went from 2045 to 1297 opens, from 1336 to 90 file status calls and from 21.6 MB to 13.7 MB read, with identical query results. #18419 (HFile block cache) also edits `HFileReaderFactory`; its production change merges cleanly with this one and only `TestHFileReaderFactory` conflicts textually. ### Risk Level low: readers get the same immutable bytes, each through its own stream, and files above the cache threshold are still streamed per reader. A reused `files` partition base file reader (`hoodie.metadata.reuse`) now keeps its downloaded file, up to `hoodie.metadata.file.cache.max.size.mb` (50 MB by default), per `HoodieBackedTableMetadata` until reset or close. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
