matanper opened a new issue, #68892: URL: https://github.com/apache/doris/issues/68892
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode); the code is unchanged on master. ### What's Wrong? `BlockFileCache::find_evict_candidates` (`be/src/io/cache/block_file_cache.cpp`) walks the LRU queue under the global `cache_lock`. It skips entries that are not releasable (in use or held) and stops only when enough space is freed or the queue ends. There is no bound on how many entries one call examines, and skipped in-use entries stay at the head of the queue. When the cache is full and many blocks are held (heavy concurrent reads plus write-through), each reservation re-walks the same in-use entries at the LRU head while holding the global lock. Lock hold time grows with the number of pinned entries, and every cache operation (reads, writes, reservations) queues behind it. We measured cache-lock waits averaging hundreds of ms (p99 in seconds) under heavy load. Upload threads that copy into the cache queued on the lock, and Stream Loads stalled. ### What You Expected? Eviction under the global lock should do bounded work per call. ### How to Reproduce? Fill the file cache to capacity, then run sustained concurrent ingestion with `enable_file_cache_write_from_s3_file_writer=true` (write-through), plus compaction or queries keeping many blocks held. Watch cache lock wait time and per-reservation scan length. ### Anything Else? What fixed it for us: 1. Bound the entries scanned per call (a mutable config, e.g. 4096). On truncation, give up on this reservation (SKIP_CACHE) instead of scanning on. 2. Move skipped in-use entries to the tail of their queue, so the next eviction doesn't re-examine them. 3. Add bvars for scanned, rotated and truncated counts. With this, lock waits dropped from seconds to about 80 µs on average under the same load. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
