RoanHeNaN commented on PR #70738: URL: https://github.com/apache/doris/pull/70738#issuecomment-7001852588
Thanks for the fix. One concern about the ANN path: **deterministic load failures (e.g. the index simply needs more memory than the BE can give) now turn into a retry on every query, with no backoff.** **Before this PR:** `_load_index_once` retains the failure, `try_load_index()` returns false, and every later query goes straight to brute force with zero load cost. That is bad for transient failures (which this PR fixes), but cheap for permanent ones. **After this PR:** the failure is recorded in `AnnIndexReader::_healthy_status`, so the next `ColumnReaderCache::_lookup` evicts the whole `ColumnReader` and rebuilds it. Then `load_index()` runs again: it re-reads the full index file (possibly from remote storage in cloud mode), `faiss::read_index` allocates a large amount of memory, fails, and frees it, and the query falls back to brute force. This repeats on every query. Effects: - Memory spikes on every query, in a situation that is already memory-constrained. This can trigger memory GC, cancel unrelated queries, or push the process toward OOM. - Read IO amplification on the index file. - Latency becomes a failed load plus brute force, which is worse than brute force alone. - Other already-loaded state on the same column (ordinal, zone map, other indexes) is discarded with the reader. - `ann_index_load_cnt` and the warning log grow per query. Suggestions (any of these would address it): 1. Back off or cap retries. For example, record the failure time or consecutive failure count, and keep the unhealthy reader (brute force, no reload) until a cooldown expires. 2. Classify errors. Retire the reader only for clearly transient errors (IO and network). For memory errors (`MEM_ALLOC_FAILED`, `MEM_LIMIT_EXCEEDED`, `std::bad_alloc`), keep the old behavior or use the backoff above. 3. Optionally, estimate the index memory from the index file size before loading. If it clearly cannot fit, skip the load and fall back without marking the reader unhealthy. A related, pre-existing issue (not introduced here, but relevant to the memory case): `AnnIndexReader::load_index` only catches `CLuceneError`, and `try_load_index()` has no try/catch. If faiss throws (`std::bad_alloc` from its own allocations, or `faiss::FaissException`), the exception goes past `try_load_index`, so the query fails instead of falling back. `DorisCallOnce` also rethrows it on every call. As far as I can tell, faiss allocations do not go through Doris's `Allocator`, so memory pressure may not show up as a clean `Status`. A UT that injects a throwing load (bad_alloc) for the ANN path would make the intended behavior explicit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
