mrhhsg opened a new pull request, #66838:
URL: https://github.com/apache/doris/pull/66838
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
`ScannerContext` counts every scanner submitted to `TaskExecutor` as
in-flight, but `TaskExecutor` admits only the TaskHandle target concurrency and
keeps excess splits in a private per-handle queue. A non-EOS admitted scanner
can then be parked by `ScannerContext` while it still owns its TaskHandle slot.
If all admitted scanners reach this state, the private backlog cannot be
admitted and no runnable scanner remains to wake the scan dependency.
A deterministic real-`TaskExecutor` unit test with four scanners and a
TaskHandle limit of two reproduces the liveness failure: before the fix, only
two scanners produce rows within five seconds, the context does not reach EOS,
and two tasks remain pending/in-flight. This change caps `ScannerContext`
concurrency at the TaskHandle initial admission target and configured per-task
cap, keeping scanner ownership consistent across the two schedulers. With the
fix, all four scanners finish and the context reaches EOS in about 11 ms.
### Release note
Fix FileScan queries that could stop making progress under TaskExecutor
concurrency throttling.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- `ENABLE_UNITY_BUILD=OFF ./run-be-ut.sh --run
--filter='ScannerContextTest.*' -j 32` (31 passed, ASAN UT)
- `build-support/clang-format.sh`
- `build-support/check-format.sh`
- `git diff --check`
- Focused clang-tidy reported no diagnostics on added lines, but the
tool exits on the existing unmatched `NOLINTEND` in `be/src/core/types.h:576`.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. FileScan scanner submission concurrency now respects
TaskExecutor per-task admission capacity.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
