airborne12 opened a new pull request, #69172:
URL: https://github.com/apache/doris/pull/69172
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68758
Problem Summary:
An unscored AND of terms on one field, such as MATCH_ALL, lists the rarest
term's rows into an array and narrows it by the other terms. A term held by
every row cannot remove any row, but the chain still reads its postings. For
`MATCH_ALL 'http status'`, where both terms are held by every row, it copies
every row id twice and rebuilds the result bitmap from the array. On a 2M-row
V2 segment the index filter took 14.4 ms, about 10 times what it took before
#68758.
A term whose document frequency equals the source's document count is held
by every row the source lists, so it cannot narrow the conjunction. The chain
now skips such terms, and a conjunction whose terms are all held by every row
returns every row, or every candidate, without reading postings. Scored queries
and conjunctions without such a term run as before.
Index filter time of MATCH_ALL on a 2M-row V2 segment, as the median ratio
of 8 paired runs on fresh backends; each column is a separate comparison and
lower is faster:
| Query | Rows holding each term | vs #68758 | vs before #68758 |
|---|---|---|---|
| `http status` | 100%, 100% | 0.01 (14.43 → 0.15 ms) | 0.11 (1.53 → 0.16
ms) |
| `rare http` | 0.05%, 100% | 0.34 (0.62 → 0.21 ms) | 0.11 (2.24 → 0.25 ms) |
| `sparse http` | 0.5%, 100% | 0.24 (1.18 → 0.29 ms) | 0.31 (1.04 → 0.34 ms)
|
| `get http` | 90%, 100% | 0.88 (14.15 → 12.20 ms) | 1.22 (10.25 → 12.83 ms)
|
| `alpha beta` | 50%, 50% | 1.02 | 1.36 |
The end-to-end time of `MATCH_ALL 'http status'` falls to 0.64 of #68758, in
line with that before #68758 (0.96). On SNII segments, which take the same
path, the `http status` filter drops from 13.67 ms to 0.37 ms, and no dense
query is slower. Two-term MATCH_ALL queries on short text and the 18 queries
used to accept #68758 (MATCH_PHRASE, MATCH_ANY and SEARCH phrases on V2 and
SNII) stay between 0.97 and 1.03 of #68758, and every 95% interval includes 1.
Dense conjunctions without a term held by every row, such as `alpha beta`, are
not changed by this PR.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Unit test: `ListedTermsTest.AConjunctionSkipsATermHeldByEveryRow` and
`ListedTermsTest.AConjunctionOfTermsHeldByEveryRowHoldsEveryRow` fail before
the change, because the chain read the term held by every row, and pass after
it.
Manual test: a 2M-row corpus with terms held by 100%, 90%, 50%, 0.5% and
0.05% of the rows was loaded into a V2 table and an SNII table. Each of 8 pairs
started two fresh backends, one per build, swapping which backend ran which
build on alternate pairs; each query ran 4 warm-up and 16 measured times on
each build, alternating between the builds. Every result matched aggregates
computed independently from the corpus.
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
