Performing verification for Oracular. I didn't have the same real-world testbed for oracular, so I just set up a lab system and enabled MGLRU and left it running stress-ng with a memory stressor for 3 days.
Its still up after 3 days, and kswapd processes are at 0%, so we will call this smoke tested, and likely fixes the issue. Marking as verified for Oracular. ** Tags removed: verification-needed-oracular-linux ** Tags added: verification-done-oracular-linux -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2087886 Title: MGLRU: kswapd uses 100% CPU when MGLRU is enabled and under memory pressure Status in linux package in Ubuntu: Fix Released Status in linux source package in Noble: Fix Committed Status in linux source package in Oracular: Fix Committed Bug description: BugLink: https://bugs.launchpad.net/bugs/2087886 [Impact] On systems with MGLRU enabled, which it is by default, if the system is under memory pressure, and some pages are then allocated, such that it wakes up kswapd to attempt page reclaim, but the system has enough memory that kswapd doesn't get OOM killed, and then there are no pages to actually reclaim, kswapd can spin at 100% endlessly, causing severe performance issues. What's happening is that lru_gen_shrink_node() unconditionally clears kswapd_failures, which can prevent kswapd from sleeping and cause 100% kswapd cpu usage even when kswapd repeatedly fails to make progress in reclaim. A workaround is to disable MGLRU, and the issue does not occur. [Fix] The fix is to only clear kswap_failures in lru_gen_shrink_node() if reclaim makes some progress, similar to shrink_node(). This was fixed in 6.12-rc4 by the commit: commit b130ba4a6259f6b64d8af15e9e7ab1e912bcb7ad Author: Wei Xu <weix...@google.com> Date: Mon Oct 14 22:12:11 2024 +0000 Subject: mm/mglru: only clear kswapd_failures if reclaimable Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b130ba4a6259f6b64d8af15e9e7ab1e912bcb7ad Both Noble and Oracular need this fix. [Testcase] The systems with this issue are DPDK compute nodes running OpenStack Yoga on Jammy. If you leave them for several days, they will hit this issue and their kswap processes will go to 100% and never drop. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 846 root 20 0 0 0 0 R 100.0 0.0 1915:16 kswapd0 846 root 20 0 0 0 0 R 100.0 0.0 1915:16 kswapd0 846 root 20 0 0 0 0 R 100.0 0.0 1915:17 kswapd0 846 root 20 0 0 0 0 R 99.0 0.0 1915:18 kswapd0 If you disable MGLRU however, they drop to 0% and sleep immediately. There is a test kernel available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf390959-test If you install the test kernel, the system should remain stable, and kswapd will not go to 100% cpu after some time. [Where problems could occur] We are adding a small requirement to clearing kswapd_failures, that we actually reclaim some memory. If we do actually reclaim some memory, the behaviour is the same as what we have now, we clear kswapd_failures and continue. If we don't manage to reclaim any memory, we leave kswapd_failures as is, and let kswapd sleep, to try again at a later time. If a regression were to occur, it would affect MGLRU users, which is the default. A regression would look like issues with kswapd memory reclaim, or lead to high cpu usage with kswapd. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2087886/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
