RoanHeNaN commented on PR #70738:
URL: https://github.com/apache/doris/pull/70738#issuecomment-7001852588

   Thanks for the fix. One concern about the ANN path: **deterministic load 
failures (e.g. the index simply needs more memory than the BE can give) now 
turn into a retry on every query, with no backoff.**
   
   **Before this PR:** `_load_index_once` retains the failure, 
`try_load_index()` returns false, and every later query goes straight to brute 
force with zero load cost. That is bad for transient failures (which this PR 
fixes), but cheap for permanent ones.
   
   **After this PR:** the failure is recorded in 
`AnnIndexReader::_healthy_status`, so the next `ColumnReaderCache::_lookup` 
evicts the whole `ColumnReader` and rebuilds it. Then `load_index()` runs 
again: it re-reads the full index file (possibly from remote storage in cloud 
mode), `faiss::read_index` allocates a large amount of memory, fails, and frees 
it, and the query falls back to brute force. This repeats on every query. 
Effects:
   
   - Memory spikes on every query, in a situation that is already 
memory-constrained. This can trigger memory GC, cancel unrelated queries, or 
push the process toward OOM.
   - Read IO amplification on the index file.
   - Latency becomes a failed load plus brute force, which is worse than brute 
force alone.
   - Other already-loaded state on the same column (ordinal, zone map, other 
indexes) is discarded with the reader.
   - `ann_index_load_cnt` and the warning log grow per query.
   
   Suggestions (any of these would address it):
   
   1. Back off or cap retries. For example, record the failure time or 
consecutive failure count, and keep the unhealthy reader (brute force, no 
reload) until a cooldown expires.
   2. Classify errors. Retire the reader only for clearly transient errors (IO 
and network). For memory errors (`MEM_ALLOC_FAILED`, `MEM_LIMIT_EXCEEDED`, 
`std::bad_alloc`), keep the old behavior or use the backoff above.
   3. Optionally, estimate the index memory from the index file size before 
loading. If it clearly cannot fit, skip the load and fall back without marking 
the reader unhealthy.
   
   A related, pre-existing issue (not introduced here, but relevant to the 
memory case): `AnnIndexReader::load_index` only catches `CLuceneError`, and 
`try_load_index()` has no try/catch. If faiss throws (`std::bad_alloc` from its 
own allocations, or `faiss::FaissException`), the exception goes past 
`try_load_index`, so the query fails instead of falling back. `DorisCallOnce` 
also rethrows it on every call. As far as I can tell, faiss allocations do not 
go through Doris's `Allocator`, so memory pressure may not show up as a clean 
`Status`. A UT that injects a throwing load (bad_alloc) for the ANN path would 
make the intended behavior explicit.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to