matanper opened a new issue, #68975: URL: https://github.com/apache/doris/issues/68975
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode, arm64). master is the same: - `be/src/common/daemon.cpp:268-301` (`je_reset_dirty_decay`): when process memory exceeds the soft limit, it sets `je_enable_dirty_page = false`. It sets it back to true only once process memory is 10% of the soft limit below it, i.e. at ~81% of `mem_limit` with the default `soft_mem_limit_frac = 0.9`. - `be/src/common/daemon.cpp:502-506` (`Daemon::je_reset_dirty_decay_thread`): while dirty pages are disabled, it sets every arena's `dirty_decay_ms` to 0, and otherwise to `je_dirty_decay_ms` (5000). ### Description Above the soft memory limit, every arena's `dirty_decay_ms` is set to 0. In jemalloc 5.3 that makes the freeing thread purge synchronously: `free` → `arena_handle_deferred_work` → `pac_decay_to_limit` → `pages_purge_forced` → `madvise(MADV_DONTNEED)`. The purge is serialized on the arena's decay mutex, and the next allocation page-faults the memory back in. Under heavy concurrent load this turns `free()` into a contention point. What we measured: - In production, during heavy concurrent loads above the soft limit, an off-CPU profile showed BE threads spending ~150 s per 2 s window blocked in `free()`. - A standalone benchmark (Release build, 96-vCPU arm64, stock `JEMALLOC_CONF`, 64 column-building workers plus 16 page-faulting threads): | Regime | ops/s | Total time blocked in free | Max free | RSS peak | |---|---|---|---|---| | Normal decay (5000 ms) | 64.5k | 7.7 s | 16 ms | 26 GB | | decay = 0 (what Doris sets over the soft limit) | 5.5k | 614 s | 81 ms | 10 GB | | decay = 1000 ms, plus `arena.<all>.decay` every 500 ms from a background thread | 63.0k | 7.7 s | 15 ms | 22 GB | At 192 workers, decay = 0 gave 9.2k ops/s, with a single `free()` taking up to 2.6 s. The hysteresis makes this sticky. Dirty decay comes back only at ~81% of `mem_limit`. If load back-pressure or the workload keeps memory between 81% and 90%, a BE that crossed the soft limit once can stay in the decay = 0 state for a long time. ### Solution Add a mutable config, e.g. `je_dirty_decay_ms_over_soft_limit` (default 0 = current behaviour). When it is greater than 0, use that decay instead of 0 above the soft limit, and have the daemon run `arena.<all>.decay` every `memory_gc_sleep_time_ms`. Memory is still returned quickly, but the purge runs on one background thread instead of on every thread that frees. In the benchmark, 1000 ms kept throughput and `free()` latency at the normal-decay level. ### How to Reproduce? Run a workload with many threads that allocate and free large buffers (heavy concurrent loads are enough) until process memory exceeds the soft limit (90% of `mem_limit`). Take an off-CPU profile: threads block in `free` → `pac_decay_to_limit` / `madvise`. Lowering `soft_mem_limit_frac` makes this easier to trigger. Changing `JEMALLOC_CONF` alone does not help, because Doris overrides each arena's decay at runtime. ### Anything Else? `enable_je_purge_dirty_pages = false` (mutable) turns the whole switch off, but then the soft limit leaves no extra pressure on dirty pages. The proposed config keeps the purge, just off the freeing threads. The trade-off is that above the soft limit, about one decay period of freed memory stays resident before it is returned. Load back-pressure and the memory GC still bound RSS. The benchmark numbers are from the benchmark build; we have not yet run the change inside a BE. Related background: #46136. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
