BiteTheDDDDt opened a new pull request, #65639:
URL: https://github.com/apache/doris/pull/65639
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Lazy materialization can turn the second-phase rowid reads into many small
and fragmented local-file reads. Prioritizing cached pages alone is not
sufficient because the remaining misses can still lose offset locality and keep
a high-IOPS random-read shape.
This PR adds an experimental, session-gated cache-aware scheduler for local
scalar lazy reads:
- add `experimental_enable_cache_aware_lazy_read`, disabled by default, and
forward it through `TQueryOptions`;
- collect and deduplicate the data pages required by the selected rowids;
- pin pages already resident in the Doris storage page cache;
- on Linux, probe the OS page cache with `preadv2(RWF_NOWAIT)` and preload
resident pages before cold reads;
- sort non-resident pages by file offset, merge nearby pages into ranges (up
to 64 KiB gap, 1 MiB span, and 1.25x read amplification), and issue
`readahead()` for those ranges;
- fall back to the existing read path when probing or prefetching is
unsupported or fails;
- expose profile counters for planned pages, Doris/OS cache hits, cold
pages, unsupported probes, prefetch ranges/bytes, and scheduling time.
The POC currently applies only to local files, query readers, scalar lazy
columns, and scans with the Doris page cache enabled. It is default-off and
does not change existing behavior unless the session variable is enabled.
Performance results are intentionally not claimed in this PR yet; the added
counters are intended to support follow-up benchmark validation and threshold
tuning.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Validation on the latest `upstream/opt_perf_4.1`:
```text
ninja -C be/build_Release -j 96 \
src/io/CMakeFiles/IO.dir/fs/local_file_reader.cpp.o \
src/exec/CMakeFiles/Exec.dir/operator/olap_scan_operator.cpp.o \
src/exec/CMakeFiles/Exec.dir/scan/olap_scanner.cpp.o \
src/storage/CMakeFiles/Storage.dir/segment/column_reader.cpp.o \
src/storage/CMakeFiles/Storage.dir/segment/segment_iterator.cpp.o
mvn -pl fe-core -am test \
-Dcheckstyle.skip=true \
-DfailIfNoTests=false \
-Dtest=org.apache.doris.qe.VariableMgrTest#testCacheAwareLazyReadSessionVariableToThrift
```
The five BE translation units compiled successfully. The FE test ran 1 test
with 0 failures and 0 errors, and the 13-module Maven reactor finished
successfully.
- Behavior changed:
- [ ] No.
- [x] Yes. Only when `experimental_enable_cache_aware_lazy_read` is
explicitly enabled; the default remains off.
- Does this need documentation?
- [x] No. This is a default-off experimental POC.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
