yiguolei opened a new pull request, #70788:
URL: https://github.com/apache/doris/pull/70788

   ### Background
   
   ANN indexes accelerate vector TopN and range predicates. A segment without 
the requested ANN index can still be evaluated by scanning its vector data.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Enhancement
   
   Problem Summary:
   
   ANN queries currently turn every returned index-loading error into 
brute-force fallback, hiding failures such as I/O and memory errors. Restrict 
fallback to a missing index and propagate other loading errors to the query 
caller. After an upgrade, queries that previously hid these failures report 
errors instead. Most added lines cover the missing-index and failure paths in 
unit tests.
   
   ### Details
   
   - **Before and after:** Previously, the boolean loading API treated every 
non-OK status as a reason to fall back. A single Status-returning `load_index` 
API now lets TopN and range callers fall back only for 
`INVERTED_INDEX_FILE_NOT_FOUND`; other statuses propagate. Missing required 
files inside an existing ANN directory remain errors.
   - **Configs and defaults:** None. The behavior applies whenever these ANN 
loading paths execute, without a new switch.
   - **Impact on existing users:** Queries may now fail on index-loading errors 
that were previously hidden by fallback. Missing-index fallback, 
small-candidate selection, persisted index formats and query syntax are 
unchanged.
   - **Behavior change:** Non-missing index-loading failures are reported to 
callers instead of silently selecting brute force.
   
   ### Testing
   
   Ran on a Linux development host with ASAN. The final commit has the same Git 
tree as the tested snapshot.
   
   ```bash
   ./run-be-ut.sh --run 
--filter='*AnnIndexLoadTest*:*AnnIndexReaderTest*:*AnnIndexIteratorTest*:VectorSearchTest.*'
 -j 16
   ```
   
   - All 124 tests from 5 suites passed, including 34 new `AnnIndexLoadTest` 
cases.
   - New cases cover init/open status propagation, repeated failed loads, 
TopN/range fallback versus errors, null-reader rejection, and missing internal 
`ann.faiss` files in HNSW, IVF and IVF_ON_DISK indexes.
   - Regression suites were not run.
   
   ### Notes for Reviewers
   
   - Normalize `INVERTED_INDEX_SNII_NOT_FOUND` only at the logical-index open 
lookup. Missing internal Faiss files must not be mistaken for an absent index.
   - Removing the `BE_TEST` loading shortcut lets tests exercise the real 
loader. Existing search-only mocks explicitly model successful loading instead.
   
   ### Release note
   
   ANN queries continue to use brute-force evaluation when a segment's ANN 
index is absent, but now report other index-loading failures instead of 
silently falling back.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes (plan: document missing-index-only 
fallback and propagation of other loading errors in apache/doris-website)
   - Pre-submission status: Partially verified: the selected ANN BE unit tests 
passed; end-to-end regression suites were not run.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Implementation, tests and this description.
   - Human review: Not stated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to