yihua opened a new issue, #20258: URL: https://github.com/apache/hudi/issues/20258
### Task Description **What needs to be done:** On table version 10 the metadata table writes native HFile log files. Every read of one of these files opens it 3 times and downloads the whole file 3 times: once for the footer (block header), once when the reader loads its schema and meta info, and once for the record or key iterator. It also makes 2 file status calls when the log file size is known from the listing, and 4 when it is not (MDT compaction builds log files from the plan's file names). Each read should download the file once per reader and reuse the length the file slice listing already returned. **Why this task is needed:** This cost applies to every metadata table read path that touches log files: writer index lookups, column stats and partition stats updates, data skipping on the driver, the timeline server and MDT compaction. It scales with lookups x log files, not with the number of keys, and on object storage every extra open and status call is a round trip. In a local Spark harness (COW table, version 10, SIMPLE index, column stats + partition stats, 9 writes + 12 reads), calls on metadata table data files totalled 2045 opens, 1336 file status calls and 21.6 MB read. In one upsert, 8 executor tasks reading 6 `column_stats` log files made 144 opens and 98 status calls. ### Task Type Performance optimization ### Related Issues **Parent feature issue:** #20257 **Related issues:** #18419 (HFile block cache) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
