airborne12 opened a new pull request, #70593:
URL: https://github.com/apache/doris/pull/70593

   ### Background
   
   SNII keeps the term dictionary in compressed blocks of about 64 KiB. An 
expansion that must find a text anywhere in a term (the contains or suffix 
token of `MATCH_PHRASE_EDGE`, a leading-wildcard glob, an unanchored regexp) 
fetches, decompresses and scans every block.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #70373, #70592 (name the literal of globs and regexps, which 
this PR then prunes on)
   PR Type: Performance
   
   Problem Summary:
   
   `MATCH_PHRASE_EDGE` and other contains or suffix expansions walked the whole 
SNII dictionary: 3-5 CPU-seconds per query on 100M rows, and about 95 s and 
10,000 S3 GETs per query on 10M rows with no file cache. This PR writes a 
per-index DICT gram summary (the blocks holding each 3-byte gram) and opens 
only the blocks holding every gram of the literal.
   
   ### Details
   
   - **Change map:** about 800 lines of production code (section format and 
reader, writer collection and budgets, read path, one proto field, one config) 
and 700 lines of tests.
   - **Why it was slow:** the key test ran on every key of every block.
   - **Optimization:** a gram-major section; a query reads its grams' rows 
only, and a gram no key holds opens no block. A text under three bytes, a 
missing section or an unknown version walks the dictionary as before.
   - **Results:** 100M rows, caches off: single-token edge 3.0-3.5 to 0.11-0.30 
CPU-seconds, multi-token 3.1-4.9 to 0.28-1.71, 32 clients 58 to 298 QPS (V3 
48). S3 with no file cache, 10M rows: 91-101 s to 7-14 s, GETs about 10,000 to 
650-1,150. Identical results.
   - **Applicability and cost:** texts of three bytes or more. Table +0.4-1.5%, 
load CPU within noise, compaction index merge about +8% CPU. The writer drops 
the summary where it cannot help or cannot be afforded; 
`snii_dict_gram_summary` (default true) turns it off.
   
   ### Testing
   
   BE UT `SniiDictGramSummary*`, `SniiDictGramSummaryWalk*`, 
`SniiCoreMetadata*`, `SniiProtoFieldNumbers*` and the SNII family; regression 
`test_storage_format_snii_gram_summary` (`MATCH_PHRASE_EDGE` and globs, SNII 
against V3); clang-tidy and the hygiene gate.
   
   ### Notes for Reviewers
   
   `SniiSectionRefsPB` gets an optional field 6, unused on branch-4.2 (older 
branches have no snii.proto); readers without it ignore the section.
   
   ### Release note
   
   SNII indexes skip the dictionary blocks that cannot hold a MATCH_PHRASE_EDGE 
text, which makes such queries several times faster, hot and on object storage.
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: No
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: Claude Code (Claude Opus 5.5)
   - Generated: code, tests and this description
   - Human review: design and benchmark results reviewed by the author; code 
review pending
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to