924060929 opened a new pull request, #69300:
URL: https://github.com/apache/doris/pull/69300
### What problem does this PR solve?
Issue Number: None
Problem Summary: External small-file scans repeatedly initialize the same
Parquet profile counter hierarchy for each file, contending on the shared
RuntimeProfile registration lock. FileReader::init also repeats the concrete
reader's initialization. Initialize counters once per scan instance and pass
the immutable handles through table-reader wrappers to Parquet readers. Keep
distinct scan instances separate. Group the shared Variant counters into one
retained bundle so deferred materialization can safely outlive the scanner
without copying a shared_ptr for every counter.
This change applies to native Parquet reads across Hive, Iceberg, Hudi,
Paimon and Fluss table-reader wrappers. It does not add speculative I/O or
change the file reader's selection and decoding behavior.
### Release note
Reduce profile initialization and counter ownership overhead when scanning
many small Parquet files.
### Check List (For Author)
- Test: Unit Test and Manual Test
- New profile-cache tests cover concurrent initialization, counter
aggregation, independent scan instances, no-profile readers and retained
Variant counter lifetime
- Reader tests cover one initialization per file and sharing across files
- run-be-ut.sh: 585 tests across 13 Parquet, ORC, Variant and
Hive/Iceberg/Hudi/Paimon/table-reader suites passed with ASAN
- Fresh Linux AArch64 build.sh --be -j3 baseline and optimized RELEASE
builds passed
- clang-format16 and header hygiene passed; preset clang-tidy passed all
19 changed files against the matching ASAN compilation database
- Isolated Release SQL results and a comparison with identical FE source
are reported below
- Behavior changed: No; preserve results and profile counter semantics
- Does this need documentation: No
### Performance validation
Complete SQL medians in milliseconds, matching Linux AArch64 Release BE,
single backend, 6 CPUs, warm footer/data cache. Both settings are fixed:
`split_batch_size=1`, `split_queue_capacity=4096`. Each single-query result has
40 measured samples in forward/reverse order after identical warmup. Cold
denotes catalog/manifest cache invalidation, not a cold JVM or BE data cache.
COUNT and SUM results are checked, and post-timing profiles verify footer/page
hits.
To isolate the BE change from a pre-existing FE EOF polling race, both
columns below use exactly the same FE source containing its notification fix;
only the Release BE package changes.
| Warm full-scan scenario | Original BE, same fixed FE | This BE, same fixed
FE | Reduction |
|---|---:|---:|---:|
| Hive 1,000 partitions | 60.77 | 16.39 | 73.0% |
| Hive 10,000 partitions | 738.04 | 120.15 | 83.7% |
| Hive one partition / 10,000 files | 495.15 | 74.42 | 85.0% |
| Iceberg 1,000 manifests / 10,000 files | 503.25 | 81.46 | 83.8% |
On unchanged baseline FE, independent BE-only single-query medians were
22.89 ms for Hive 1,000 partitions, 230.37 ms for Hive 10,000 partitions,
117.95 ms for Hive one partition with 10,000 files and 138.38 ms for Iceberg
1,000 manifests. The faster BE exposed an existing FE terminal-queue poll: 78
delayed empty requests in a 400-SQL diagnostic; one finished production at
0.148 ms but returned at 100.911 ms. The companion FE scan/scheduling PR fixes
terminal notification, verified with zero such delays on replay. Merge the FE
notification fix first when assessing concurrent latency; the BE patch has no
code dependency on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
