yihua opened a new issue, #20257: URL: https://github.com/apache/hudi/issues/20257
### Task Description **What needs to be done:** On object storage every file-system call (HEAD, GET, LIST, PUT) is a round trip, and every executor request to the embedded timeline server is a driver round trip served under a lock. Small redundant costs per file, file group, task, partition or instant add up to the dominant latency on large tables, and they are invisible in tests that use a handful of files on one thread. An audit of master instrumented every storage call (and every executor to timeline-server request) made by Spark writes, table services, queries, SQL DMLs and HoodieStreamer, across COW and MOR, table versions 6, 8 and current, the metadata table on and off, the timeline server on and off, the index types and the write handles. It found redundant calls such as: metadata-table HFiles opened and downloaded three times per read, base-file footers read twice and status-checked twice per file, column stats re-reading every new file's footer, executor tasks rebuilding metadata readers and file-system views, per-task timeline-server lookups the driver could answer once, and a per-request lock plus timeline hash in the timeline server. It also surfaced several correctness bugs. This issue tracks the fixes. Each sub-issue describes one problem with its evidence; each PR carries a test that counts the calls (or reproduces the bug) so the regression class is guarded. **Why this task is needed:** On object storage each redundant call is a 10-100 ms round trip, and the costs found grow with files, file groups, tasks, partitions or instants, so they dominate latency on large tables. ### Task Type Performance optimization ### Related Issues **Parent feature issue:** (none) **Related issues:** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
