yihua opened a new issue, #20262: URL: https://github.com/apache/hudi/issues/20262
### Task Description **What needs to be done:** Let the bloom index key-check stage use the base files the driver already resolved while loading key ranges, instead of looking each one up again on the executors, and let the BLOOM and SIMPLE index footer and key reads open parquet base files from their known size instead of looking up the file status first. **Why this task is needed:** The key check builds a `HoodieKeyLookupHandle` per candidate file group, and the handle resolves the latest base file through the file system view twice (once for the bloom filter, once in `getLookupResult`). On executors that is the remote timeline-server view, so every (task, candidate base file) costs two `getLatestBaseFile` requests, each a per-partition scan under the view read lock, for every BLOOM / GLOBAL_BLOOM upsert or delete with the timeline server on (both defaults). The count grows linearly with file groups per partition. Each parquet open also stats the file first (a HEAD on object storage) although its size is already known from the listing or the metadata table. On a local Spark COW upsert harness, going from 1 to 4 file groups per partition grows executor timeline-server requests from `HoodieBloomIndexCheckFunction` from 64 to 256 (driver requests stay flat), and executor base-file `getFileStatus` calls from 152 to 352. The two lookups per handle date from #726 (0.5.0) and became timeline-server requests when the embedded timeline server became the default (#1634, 0.5.3). #6793 (HUDI-4917, 0.13.0) already reuses the driver-resolved base file in the range-info stage, but not in the key check. ### Task Type Performance optimization ### Related Issues **Parent feature issue:** #20257 **Related issues:** - #6793 (driver-resolved base file for the range-info stage) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
