airborne12 opened a new pull request, #69909:
URL: https://github.com/apache/doris/pull/69909
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: An equality filter reads a posting list of millions of
documents through
`decode_window_prefix_plan`, so small costs per document, per PFOR run or
per window add up. In the
stack samples of a filtered phrase prefix query on a 99M-row table:
- The window loop cleared its vector, so every window zero-filled it again
before decoding wrote over
it (4% of the SNII samples).
- The sum of the docid deltas checked for a zero delta and for overflow at
every document (about 7%,
with the sum itself).
- Each PFOR run read its exception count, and each exception its two fields,
through the out-of-line
varint decoders, although they are nearly always one byte (about 5%).
The window loop now decodes over the vector, so a resize zero-fills only
what it adds. A window's
deltas are checked whole, with no branch per document, before they are
summed: the same checks, and
the same error code when one fails. If a window breaks more than one of
them, the zero delta is
reported first, whichever came first in the window. `pfor_decode` and
`pfor_skip` use
`get_varint32_fast`, which handles a one-byte value inline and leaves
anything else to the checked
decoder. The on-disk format and the write path do not change.
| Query (99M rows), new/previous | Hot | Lowest run, new/previous | Cold |
|---|---|---|---|
| filtered phrase prefix | 0.94 / 0.96 / 0.86 | 0.95 / 0.93 / 0.93 | 0.98 |
| phrase with tenant, app, type and status | 0.92 / 0.92 / 0.98 | 0.92 /
0.95 / 0.95 | 0.95 |
| phrase with a latency filter | 0.95 / 0.91 / 0.99 | 0.94 / 0.92 / 0.96 | |
| MATCH_ALL with the same four filters | 0.94 / 0.92 / 1.10 | 0.94 / 0.93 /
0.93 | |
| top-50 with tenant and payload equalities | 0.94 / 0.94 / 0.95 | 0.95 /
0.94 / 0.99 | 1.00 |
| the other queries | 0.93-1.03 | | |
Hot: both versions interleaved in one BE process on a quiet host, three runs
each. Cold: BE restarted
and the index files evicted before every run, configuration order
alternated, 8 runs for the phrase
prefix, 5 for the others. The build also had #69879, #69882, #69887, #69891
and #69895.
### Release note
Speed up SNII queries that filter on values matching many rows.
### Check List (For Author)
- Test: Unit Test (`SniiFrqPod`: a zero delta in every position of a window
of three PFOR runs is
rejected and a zero first delta is not, the sum may reach the last docid
but not pass it, deltas
that add up to 2^32, which is 0 in 32 bits, are rejected, and a window
decodes over a vector of
another size; six mutants of the checks all fail tests; all inverted index
tests pass under ASAN).
Manual test: the hot and cold A/B above.
- Behavior changed: No
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
