Authorlove opened a new pull request, #420:
URL: https://github.com/apache/doris-thirdparty/pull/420
### What problem does this PR solve?
Issue Number: None
Related PR: Apache Doris integration PR (to be linked)
Problem Summary:
`IndexWriter::indexCompaction` filters deleted documents through the row-ID
translation map but copies the sum of every source field's token count into the
first destination index and writes zero to the remaining destinations.
Consequently BM25 uses stale average document lengths after deletes and
incorrect per-segment statistics after a split.
A minimal Doris reproduction uses a single-bucket UNIQUE KEY table with an
English inverted index and automatic compaction disabled: retain `alpha` and
`alpha alpha` followed by 14 `omega` tokens, and insert a delete-sign row
containing 240 `alpha` plus 16 `omega` tokens. Compare `SEARCH('body:alpha')
ORDER BY score() DESC` after full compaction with a fresh table containing just
the two visible rows. For these two visible documents of lengths 1 and 16 and
one deleted document of length 256, the clean index contains 17 tokens. Native
index compaction retains a token count of 273: on Doris 4.1.5 the visible
documents score 0.3069872 / 0.33349273, while rebuilding the same documents
scores 0.28530657 / 0.20084909 and reverses their ranking.
Accumulate exact term frequencies from retained postings into each
destination field while merging terms, then write those destination totals
alongside the remapped norms. This also repairs stale source totals when their
postings still retain frequencies. It neither changes BM25's formula nor
changes visibility/statistics before compaction.
An index with norms but without term frequencies cannot supply an exact
count: byte norms are quantized. Reject native compaction for that case before
opening output writers; the companion Doris change selects its existing
raw-column index rebuild path. Indexes without norms continue to compact
without requiring frequencies.
### Release note
Correct BM25 token statistics after inverted-index compaction removes or
redistributes documents.
### Check List (For Author)
- Test
- [x] Unit Test
- [x] Manual test (pre-fix SQL reproduction on Doris 4.1.5)
- [ ] Regression test
- Native CLucene ASAN build, using the repository's CMake/`cl_test`
entrypoints.
- `cl_test -v -p IndexCompaction IndexCompress IndexCompressV3
testBlockMaxScoreV3`: 16 tests passed.
- Rebuilt the original `08d45e541fef1e48dc4ab00d2d5fde9569b49ee3`
implementation with the new tests: 2 new tests fail (expected destination token
count 42, actual 315; missing-frequency guard absent); the 4 existing
compaction tests pass.
- Coverage includes deleted long documents, a non-exact byte-norm length
(41 tokens), two source/destination indexes, multiple fields, empty field
values, and indexes with/without norms or frequencies.
- A full Doris master binary has not been built or exercised locally;
the companion PR records its integration validation separately.
- Behavior changed:
- [x] Yes. Compacted norms carry exact destination token totals. Native
compaction rejects norms-bearing indexes without frequencies; callers must
rebuild them.
- Does this need documentation?
- [x] No. Correctness fix; no new setting or on-disk format.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
