eldenmoon opened a new pull request, #68140:
URL: https://github.com/apache/doris/pull/68140
### What problem does this PR solve?
Issue Number: None
Related PR: #68039
Problem Summary:
This is a selective backport of #68039 to `branch-4.1`. It keeps the
CLucene/V2/V3 VARIANT-path
norms handling and deliberately excludes all SNII-related changes.
An inverted index on a VARIANT column is copied to every extracted
subcolumn, so dense BM25 norms
can consume substantial storage when many paths are indexed. This backport
adds the `norms` index
property and the mutable BE config `inverted_index_skip_norms_for_variant`
(off by default). The
config omits norms for VARIANT-path indexes regardless of the index property.
CLucene BM25 scoring now rejects an analyzed index when any segment was
written without norms,
while MATCH filtering remains available. This prevents NaN or misleading
scores for mixed-generation
segments. Whole-column and subcolumn VARIANT regression coverage is included.
SNII scope:
- The upstream SNII policy/compaction changes are not included.
- No files under `be/src/storage/index/snii/**` or SNII-specific
tests/regressions are changed.
- The final diff contains no SNII-related paths or additions.
### Release note
Added the `norms` inverted-index property and the BE config
`inverted_index_skip_norms_for_variant`
for CLucene/V2/V3 VARIANT-path indexes. BM25 `score()` now fails when an
analyzed segment has no
norms instead of returning invalid or inconsistent scores.
### Check List (For Author)
- Test:
- `./build.sh --fe`: BUILD SUCCESS; Checkstyle reports 0 violations.
- `./run-fe-ut.sh --run
org.apache.doris.analysis.InvertedIndexNormsPropertyTest`: 2 passed.
- clang-format check and `git diff --check`: passed.
- BE build, BE unit tests, and regression tests were not run because the
local ASAN CMake
configuration is blocked by LeakSanitizer under ptrace/OpenBLAS
`getarch`; the regression
golden was reused from the upstream non-SNII result and was not
regenerated locally.
- Behavior changed: Yes (BM25 scoring is rejected for analyzed indexes
without norms; defaults
remain unchanged).
- Does this need documentation: Yes (follow-up to the upstream documentation
work).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
