englefly opened a new pull request, #70795:
URL: https://github.com/apache/doris/pull/70795

   ### Background
   
   Scan operators collect available runtime filters during initialization and 
check for additional filters when a scanner runs. A scan local state shares its 
residual conjuncts across scanners. Each scanner clones those expressions for 
execution and refreshes its clone when its applied runtime filter count changes.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Performance
   
   Problem Summary:
   
   While a runtime filter is pending, each scanner check can repeat the same 
cost sort under the shared conjunct lock. For 32 conjuncts and 1,000 checks 
with no new expressions, an isolated count dropped from 220,000 cost calls to 
zero. Sort the initial residual expressions before the first scanner clone and 
sort again when the helper appends new expressions.
   
   ### Details
   
   - **Why it was slow:** The late-arrival path stable-sorted the full conjunct 
list before checking for new expressions. Comparing two conjuncts walks their 
expression trees even when the order is already correct.
   - **Optimization:** Stable-sort the residual list after pushdown and before 
scanner initialization. On late arrival, snapshot the newly appended 
expressions for pruning and re-sort the shared list under the existing lock. A 
disabled filter still advances the applied count.
   - **Results:** One isolated run per variant on an Intel Core i7-9800X with 
Clang 20.1.7 and ASAN. The harness ran the respective 
`update_late_arrival_runtime_filter` implementation from master at 
`5bc6ed62f9b` and the candidate with a consumer stub configured to append 
nothing. Both started with the same pre-sorted conjuncts. Across 1,000 checks, 
cost calls changed from 6,000 to 0 for 3 conjuncts and from 220,000 to 0 for 
32. A single conjunct and the disabled ordering control each made zero cost 
calls on both versions. These counts measure the late-check path in isolation.
   - **Applicability and cost:** The existing session variable 
`enable_adjust_conjunct_order_by_cost` defaults to true. Scans with runtime 
filter descriptors can pay for one initial sort when at least two residual 
conjuncts remain. Savings depend on the number of checks that find no new 
expressions.
   
   ### Testing
   
   - `build-support/check-format.sh` with clang-format 16 and 
`build-support/check-build-hygiene.sh` passed locally.
   - The ASAN BE unit build compiled `scan_operator.cpp`. The test translation 
unit passed a syntax check with the ASAN compilation flags.
   - Added scan unit cases for initial and late ordering, stable ties, disabled 
filters, peer scanner refresh, and the rows returned after applying a late IN 
filter. The local ASAN unit test run is pending.
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: No
   - Pre-submission status: Partially verified: ASAN scan tests and clang-tidy 
pending.
   
   ### AI Disclosure
   
   - Tool: OpenAI Codex (GPT-6)
   - Generated: Code, unit tests, and PR description.
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to