This makes the series operative. So far, nothing called mmput_exit().
Switch exit_mm() from mmput() to mmput_exit(), so the exiting task's final
reference drop can defer the address-space teardown to mm_reaper
instead of running exit_mmap() on the exiting CPU.
Only the exit path is routed. Every other mmput() caller (get_task_mm()
users like ptrace and /proc, kthread_unuse_mm(), etc.) still tears down
inline, and if one of those ends up holding the actual last reference,
behavior is unchanged -- the deferral applies only when the exiting
task's own drop is the final one.
The deferral triggers only when all of the following hold, and falls
back to inline __mmput() otherwise:
- vm.async_mm_teardown=1 (static key, default off)
- RSS >= vm.async_mm_teardown_thresh_pages (default 64MB)
- the mm is not an OOM target (MMF_OOM_TARGETED) and has not already
been reaped (MMF_OOM_SKIP)
- charging the RSS to the pending budget stays under
vm.async_mm_teardown_max_pending_pages
Results, bare metal, 32 CPUs / 32GB RAM (31 GiB usable), single
NUMA node, performance governor, medians of 10 reps, THP off unless noted,
vm.async_mm_teardown_max_pending_pages raised to 3/4 of RAM for the
async runs (the RAM/4 default would reject a single 16GB mm on this
box; see the cover letter's open question on the cap default):
reap latency (SIGKILL -> waitpid() returns), async off -> on:
1GB 18.01 ms -> 0.14 ms
4GB 73.71 ms -> 0.14 ms
16GB 282.16 ms -> 0.15 ms
16GB 27.68 ms -> 0.13 ms (THP on)
voluntary exit (_exit() -> reap) matches within noise:
16GB 271.27 ms -> 0.12 ms
redis-server with a 16GB populated dataset (real heap, VmRSS ~16.2GB):
292.97 ms -> 0.24 ms
32 simultaneous 0.5GB exits, tail until all reaped:
157.42 ms -> 0.32 ms
below-threshold 30MB process with the feature enabled:
0.56 ms -> 0.60 ms (sync path, within noise)
A CONFIG_ASYNC_MM_TEARDOWN=n build of the same tree reproduces the
async-off column within noise (18.01 vs 18.01 ms at 1GB, 282.14 vs
282.16 ms at 16GB), so the config itself costs nothing when dark.
The work is moved, not eliminated: with async on, the 16GB region is
fully back in the buddy allocator ~240 ms after the kill (vs ~283 ms
when torn down inline), and the reaper spends ~1.18x the inline CPU on
the same teardowns (nice-19 kthread, cache-cold on another CPU). The
cover letter carries the full matrix: freeing-latency checks,
backpressure fallback, OOM-killer interaction (an OOM victim is never
queued, tracepoint-verified, including the CLONE_VM-sharer case),
tmpfs inode eviction in reaper context, sysctl toggling under load,
and a freezer cycle with a queued backlog.
Signed-off-by: Aditya Sharma <[email protected]>
---
kernel/exit.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/exit.c b/kernel/exit.c
index 1056422bc..6f97d204f 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -607,7 +607,7 @@ static void exit_mm(void)
task_unlock(current);
mmap_read_unlock(mm);
mm_update_next_owner(mm);
- mmput(mm);
+ mmput_exit(mm);
if (test_thread_flag(TIF_MEMDIE))
exit_oom_victim();
}
--
2.34.1
