airborne12 opened a new pull request, #70593: URL: https://github.com/apache/doris/pull/70593
### Background SNII keeps the term dictionary in compressed blocks of about 64 KiB. An expansion that must find a text anywhere in a term (the contains or suffix token of `MATCH_PHRASE_EDGE`, a leading-wildcard glob, an unanchored regexp) fetches, decompresses and scans every block. ### What problem does this PR solve? Issue Number: None Related PR: #70373, #70592 (name the literal of globs and regexps, which this PR then prunes on) PR Type: Performance Problem Summary: `MATCH_PHRASE_EDGE` and other contains or suffix expansions walked the whole SNII dictionary: 3-5 CPU-seconds per query on 100M rows, and about 95 s and 10,000 S3 GETs per query on 10M rows with no file cache. This PR writes a per-index DICT gram summary (the blocks holding each 3-byte gram) and opens only the blocks holding every gram of the literal. ### Details - **Change map:** about 800 lines of production code (section format and reader, writer collection and budgets, read path, one proto field, one config) and 700 lines of tests. - **Why it was slow:** the key test ran on every key of every block. - **Optimization:** a gram-major section; a query reads its grams' rows only, and a gram no key holds opens no block. A text under three bytes, a missing section or an unknown version walks the dictionary as before. - **Results:** 100M rows, caches off: single-token edge 3.0-3.5 to 0.11-0.30 CPU-seconds, multi-token 3.1-4.9 to 0.28-1.71, 32 clients 58 to 298 QPS (V3 48). S3 with no file cache, 10M rows: 91-101 s to 7-14 s, GETs about 10,000 to 650-1,150. Identical results. - **Applicability and cost:** texts of three bytes or more. Table +0.4-1.5%, load CPU within noise, compaction index merge about +8% CPU. The writer drops the summary where it cannot help or cannot be afforded; `snii_dict_gram_summary` (default true) turns it off. ### Testing BE UT `SniiDictGramSummary*`, `SniiDictGramSummaryWalk*`, `SniiCoreMetadata*`, `SniiProtoFieldNumbers*` and the SNII family; regression `test_storage_format_snii_gram_summary` (`MATCH_PHRASE_EDGE` and globs, SNII against V3); clang-tidy and the hygiene gate. ### Notes for Reviewers `SniiSectionRefsPB` gets an optional field 6, unused on branch-4.2 (older branches have no snii.proto); readers without it ignore the section. ### Release note SNII indexes skip the dictionary blocks that cannot hold a MATCH_PHRASE_EDGE text, which makes such queries several times faster, hot and on object storage. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, tests and this description - Human review: design and benchmark results reviewed by the author; code review pending 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
