airborne12 opened a new pull request, #69648: URL: https://github.com/apache/doris/pull/69648
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: DorisSniiFileReader::read_batch submits every physical segment of a batch with more than one segment to the shared buffered-reader prefetch thread pool and waits on a latch. When the index file is in the page cache the reads take microseconds and 99% of the batches of a phrase workload are under 256 KB, so the submit, wake-up and wait cost more than the reads, and a query that issues thousands of batches from 16 scanner threads queues on the pool's lock. On 99M AgentLogs rows, 83% of the threads inside the SNII query path were blocked in ThreadPool::do_submit for the date-range phrase query. A batch of at most config::snii_inline_read_max_segments (default 4, 0 disables) segments and 256 KiB is now read on the calling thread when the index file is on the local filesystem and the moving average of recent small physical reads is below 20 us. Remote and cached-remote readers, larger batches and slow (page-cache miss) reads still fan out to the pool, because overlapping disk reads gain from it: with a fixed rule instead of the average, the cold run of two phrase queries was 4-8% slower. The serial read rounds counter counts one round per segment for a batch read inline. Measured on 99M AgentLogs rows (branch-4.2 build; the changed code is the same on master), same process, 25 queries, inline off against on, 16 samples each: date-range phrase query 0.39x, any-of-three-phrases top-N 0.70x, no query above 1.02x. Cold runs (restart, page cache evicted, three repetitions) of eight queries are within 2% of the pool. ### Release note Faster SNII phrase queries on local storage. ### Check List (For Author) - Test: Unit Test (DorisSniiFileReaderInlineReadTest: small local batch inline, large local batch, remote batch and slow reads fan out, large bytes fan out, 0 disables); Snii*, DorisSniiFileReader*, *InvertedIndex*, *IndexFileWriter* and *IndexFileReader* pass on master (1723 tests, with the zstd saving change below stacked). Manual test: interleaved hot A/B and repeated cold runs on 99M rows. - Behavior changed: No - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
