On 11/27/2025 9:14 AM, Barry Song wrote:
From: Barry Song <[email protected]>
Oven observed most mmap_lock contention and priority inversion
come from page fault retries after waiting for I/O completion.
Oven subsequently raised the following idea:
There is no need to always fall back to mmap_lock if the per-VMA
lock was released only to wait for pagecache or swapcache to
become ready.
In this case, the retry path can continue using the per-VMA lock.
This is a big win: it greatly reduces mmap_lock acquisitions.
Here are some test data for the patches:
== Test Case ==
Douyin (Chinese version of TikTok) warm start on smart phone with 8GB RAM
== mmap_lock Acquisitions And Wait Time ==
Metric Before (Avg) After (Avg) Change
------------------------------------------------------------------------
Read Lock Count 20,010 5,719 -71.42%
Read Total Wait (us) 10,695,877 408,436 -96.18%
Read Avg Wait (us) 534.00 71.00 -86.70%
Write Lock Count 838 909 +8.47%
Write Total Wait (us) 501,293 97,633 -80.52%
Write Avg Wait (us) 598.00 107.00 -82.11%
== Read Lock Waiting Time Distribution of mmap_lock ==
Range (us) Before (Avg) After (Avg) Change
------------------------------------------------------------------------
[0, 1) 9,927 4,286 -56.82%
[1, 10) 9,179 1,327 -85.54%
[10, 100) 191 88 -53.93%
[100, 1000) 57 6 -89.47%
[1000, 10000) 328 9 -97.26%
[10000, 100000) 328 6 -98.17%
[100000, 1000000) 0 0 N/A
[1000000, +) 0 0 N/A
== Write Lock Waiting Time Distribution of mmap_lock ==
Range (us) Before (Avg) After (Avg) Change
------------------------------------------------------------------------
[0, 1) 250 300 +20.00%
[1, 10) 483 556 +15.11%
[10, 100) 52 41 -21.15%
[100, 1000) 12 5 -58.33%
[1000, 10000) 22 4 -81.82%
[10000, 100000) 16 1 -93.75%
[100000, 1000000) 0 0 N/A
[1000000, +) 0 0 N/A
After the optimization, the number of read lock acquisitions is
significantly reduced, and both lock waiting time and tail latency are
dramatically improved.
Thanks,
Oven
Oven Liyang (1):
mm/filemap: Retry fault by VMA lock if the lock was released for I/O
Barry Song (1):
mm/swapin: Retry swapin by VMA lock if the lock was released for I/O
arch/arm/mm/fault.c | 5 +++++
arch/arm64/mm/fault.c | 5 +++++
arch/loongarch/mm/fault.c | 4 ++++
arch/powerpc/mm/fault.c | 5 ++++-
arch/riscv/mm/fault.c | 4 ++++
arch/s390/mm/fault.c | 4 ++++
arch/x86/mm/fault.c | 4 ++++
include/linux/mm_types.h | 9 +++++----
mm/filemap.c | 5 ++++-
mm/memory.c | 10 ++++++++--
10 files changed, 47 insertions(+), 8 deletions(-)
Cc: Russell King <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Huacai Chen <[email protected]>
Cc: WANG Xuerui <[email protected]>
Cc: Madhavan Srinivasan <[email protected]>
Cc: Michael Ellerman <[email protected]>
Cc: Nicholas Piggin <[email protected]>
Cc: Christophe Leroy <[email protected]>
Cc: Paul Walmsley <[email protected]>
Cc: Palmer Dabbelt <[email protected]>
Cc: Albert Ou <[email protected]>
Cc: Alexandre Ghiti <[email protected]>
Cc: Alexander Gordeev <[email protected]>
Cc: Gerald Schaefer <[email protected]>
Cc: Heiko Carstens <[email protected]>
Cc: Vasily Gorbik <[email protected]>
Cc: Christian Borntraeger <[email protected]>
Cc: Sven Schnelle <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Andy Lutomirski <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: [email protected]
Cc: H. Peter Anvin <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Lorenzo Stoakes <[email protected]>
Cc: Liam R. Howlett <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Mike Rapoport <[email protected]>
Cc: Suren Baghdasaryan <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: Pedro Falcato <[email protected]>
Cc: Jarkko Sakkinen <[email protected]>
Cc: Oscar Salvador <[email protected]>
Cc: Kuninori Morimoto <[email protected]>
Cc: Oven Liyang <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Ada Couprie Diaz <[email protected]>
Cc: Robin Murphy <[email protected]>
Cc: Kristina Martšenko <[email protected]>
Cc: Kevin Brodsky <[email protected]>
Cc: Yeoreum Yun <[email protected]>
Cc: Wentao Guan <[email protected]>
Cc: Thorsten Blum <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Yunhui Cui <[email protected]>
Cc: Nam Cao <[email protected]>
Cc: Chris Li <[email protected]>
Cc: Kairui Song <[email protected]>
Cc: Kemeng Shi <[email protected]>
Cc: Nhat Pham <[email protected]>
Cc: Baoquan He <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
--
2.39.3 (Apple Git-146)
