Previous description, for context: lxcfs provides a container-specific view of /proc/meminfo. Occasionally, with near zero or zero swap usage *and* swap accounting turned on (kernel parameter swapaccount=1), the container will report 100% swap utilization.
Example from Ubuntu 18.04 and lxcfs 3.0.3-0ubuntu1~18.04.1, using the output of "while true; do free |grep Swap >> swap.log; done" with the "free" command's header for clarity: total used free shared buff/cache available Mem: 2040052 53028 1839908 176 147116 1987024 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 2097148 0 <--- The issue Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Swap: 2097148 0 2097148 Additionally, "watch -n 0.1 "grep Swap /proc/meminfo"" was running in another terminal on the container, this seems to stimulate the behavior. This appears to come from proc_meminfo_read in bindings.c in lxcfs. >From the kernel documentation (https://github.com/torvalds/linux/blob/master/Documentation/admin- guide/cgroup-v1/memory.rst), under 5.5 usage_in_bytes: "For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2)." When swap accounting is on and swap usage is 0 or very close to 0, the kernel reported mem+swap usage and the mem usage numbers can cross over, causing the assignments below to set swapfree to 0, triggering the issue: swapusage = memswusage - memusage, swapfree = swapusage < swaptotal ? swaptotal - swapusage : 0; ** Description changed: - lxcfs provides a container-specific view of /proc/meminfo. Occasionally, - with near zero or zero swap usage *and* swap accounting turned on - (kernel parameter swapaccount=1), the container will report 100% swap - utilization. + [Impact] - Example from Ubuntu 18.04 and lxcfs 3.0.3-0ubuntu1~18.04.1, using the output of "while true; do free |grep Swap >> swap.log; done" with the "free" command's header for clarity: - total used free shared buff/cache available - Mem: 2040052 53028 1839908 176 147116 1987024 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 2097148 0 <--- The issue - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 - Swap: 2097148 0 2097148 + * lxcfs provides a container-specific view of /proc/meminfo. + Occasionally, with near zero or zero swap usage *and* swap accounting + turned on (kernel parameter swapaccount=1), the container will report + 100% swap utilization. - Additionally, "watch -n 0.1 "grep Swap /proc/meminfo"" was running in - another terminal on the container, this seems to stimulate the behavior. + * This issue has been encountered and could result in unecessary alerts + or potential automated attempts at remediating a non-existent "full + swap" issue. - This appears to come from proc_meminfo_read in bindings.c in lxcfs. + * This fix changed the logic used for SwapFree when swap accounting is + enabled to better handle situations where memswusage is less than + memusage, caused by the fuzziness of the usage_in_bytes counters used as + the source. Specifically, it added a check for memusage being larger + than memswusage and if so, sets 0 as the value of swapusage. Otherwise + the calculation (memswusage - memusage) remains the same. - From the kernel documentation - (https://github.com/torvalds/linux/blob/master/Documentation/admin- - guide/cgroup-v1/memory.rst), under 5.5 usage_in_bytes: + [Test Case] - "For efficiency, as other kernel components, memory cgroup uses some - optimization to avoid unnecessary cacheline false sharing. - usage_in_bytes is affected by the method and doesn't show 'exact' value - of memory (and swap) usage, it's a fuzz value for efficient access. (Of - course, when necessary, it's synchronized.) If you want to know more - exact memory usage, you should use RSS+CACHE(+SWAP) value in - memory.stat(see 5.2)." + * Requires a Bionic (18.04) or Eoan (19.10) host with swap space. - When swap accounting is on and swap usage is 0 or very close to 0, the kernel reported mem+swap usage and the mem usage numbers can cross over, causing the assignments below to set swapfree to 0, triggering the issue: - swapusage = memswusage - memusage, swapfree = swapusage < swaptotal ? swaptotal - swapusage : 0; + * Enable swap accounting with the "swapaccount=1" kernel parameter on + the kernel command line. Edit /etc/default/grub, add "swapaccount=1" to + the GRUB_CMDLINE_LINUX_DEFAULT="other parameters" line, then run + "update-grub" and reboot to make the change active. + + * Ensure lxd is installed, "sudo apt install lxd" + + * Create a lxd/lxc container with "lxc launch ubuntu:X container_name" + with X being either b[ionic] or e[oan]. + + * Open two shells to the container with "lxc shell container_name" + + * In one of the shells, run: watch -n 0.1 "grep Swap /proc/meminfo" + + * In the other, run: while true; do free; done + + * You should see SwapFree intermittently drop to zero in the first + terminal. + + * The fix results in small non-zero swap "usage" intermittently instead + of intermittent SwapFree = 0. + + + [Regression Potential] + + * Low, as swap accounting must be enabled to encounter the bug and the + fix. + + * Potential for unanticipated edge cases in the values of memusage and + memswusage to cause incorrect swap reporting within the container, with + swap accounting turned on. + + * Any tooling that expected, compensated, or relied on the behavior may + no longer work as expected. + + [Other Info] + + * Cherrypick of a one line fix to address this specific situation. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1860813 Title: LXC container reports spike in swap occasionally To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1860813/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
