Stefan, while recent kernels seem happier than previous kernels (I think -14 era was terrible), I don't think this problem is fixed yet:
sarnold@millbarge:/tmp$ uname -a Linux millbarge 5.4.0-21-generic #25-Ubuntu SMP Sat Mar 28 13:10:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux sarnold@millbarge:/tmp$ uptime 17:18:02 up 14:59, 8 users, load average: 0.36, 0.27, 0.16 sarnold@millbarge:/tmp$ time journalctl > foo real 0m47.135s user 0m36.886s sys 0m10.180s sarnold@millbarge:/tmp$ ls -lh /tmp/foo -rw-rw-r-- 1 sarnold sarnold 841M Apr 3 17:15 /tmp/foo sarnold@millbarge:/tmp$ wc -l /tmp/foo 4045530 /tmp/foo During that journalctl run, my mouse could not move for twenty seconds. I left an iovisor tool running to log high-latency __alloc_pages_nodemask() calls (identified by Sultan as a likely cause): # funcslower-bpfcc __alloc_pages_nodemask Tracing function calls slower than 1 ms... Ctrl+C to quit. COMM PID LAT(ms) RVAL FUNC dp_sync_taskq 716 7.72 fffff2d109115040 __alloc_pages_nodemask systemd-udevd 44671 1.59 fffff2d10575b800 __alloc_pages_nodemask git 46218 4.27 fffff2d10500edc0 __alloc_pages_nodemask z_wr_iss 683 4.90 fffff2d101b4cf40 __alloc_pages_nodemask systemd-udevd 51359 4.27 fffff2d10ec12800 __alloc_pages_nodemask systemd-udevd 56585 2.48 fffff2d10d3b0500 __alloc_pages_nodemask systemd-udevd 56577 2.55 fffff2d106f73380 __alloc_pages_nodemask systemd-udevd 72552 2.09 fffff2d105249680 __alloc_pages_nodemask systemd-udevd 76964 1.53 fffff2d1017fd440 __alloc_pages_nodemask systemd-udevd 80900 3.94 fffff2d102675200 __alloc_pages_nodemask systemd-udevd 88669 2.09 fffff2d1015fb5c0 __alloc_pages_nodemask kthreadd 2 4.41 fffff2d10d6f96c0 __alloc_pages_nodemask kthreadd 2 4.65 fffff2d10d38c3c0 __alloc_pages_nodemask kthreadd 2 3.97 fffff2d10d3cc780 __alloc_pages_nodemask updatedb.mloca 106194 2.44 fffff2d10e1c2680 __alloc_pages_nodemask kthreadd 2 1.20 fffff2d10d36ff40 __alloc_pages_nodemask kthreadd 2 4.13 fffff2d10d495940 __alloc_pages_nodemask kthreadd 2 6.34 fffff2d10d317180 __alloc_pages_nodemask kthreadd 2 4.76 fffff2d110e85000 __alloc_pages_nodemask systemd-udevd 113822 1.51 fffff2d10f5c8e00 __alloc_pages_nodemask systemd-udevd 113820 1.91 fffff2d10e611800 __alloc_pages_nodemask systemd-udevd 122017 3.13 fffff2d10e539700 __alloc_pages_nodemask systemd-udevd 141102 2.43 fffff2d104487200 __alloc_pages_nodemask systemd-udevd 144565 4.82 fffff2d106b84040 __alloc_pages_nodemask systemd-udevd 1156 1.28 fffff2d102325c00 __alloc_pages_nodemask systemd-udevd 152486 3.07 fffff2d101516500 __alloc_pages_nodemask systemd-udevd 152485 4.94 fffff2d1015cbbc0 __alloc_pages_nodemask systemd-udevd 172807 1.33 fffff2d1015f4700 __alloc_pages_nodemask systemd-udevd 185503 4.46 fffff2d10d321240 __alloc_pages_nodemask systemd-udevd 189244 5.12 fffff2d10e7f0b00 __alloc_pages_nodemask systemd-udevd 205973 5.36 fffff2d10d4171c0 __alloc_pages_nodemask systemd-udevd 210133 5.19 fffff2d10ffac340 __alloc_pages_nodemask systemd-udevd 209621 2.95 fffff2d10d377fc0 __alloc_pages_nodemask systemd-udevd 218768 2.24 fffff2d10526e500 __alloc_pages_nodemask systemd-udevd 234261 1.42 fffff2d101814680 __alloc_pages_nodemask systemd-udevd 235468 3.82 fffff2d10d2afa40 __alloc_pages_nodemask systemd-udevd 236826 4.94 fffff2d10d0f8440 __alloc_pages_nodemask systemd-udevd 247115 1.13 fffff2d1023ee180 __alloc_pages_nodemask systemd-udevd 247573 1.46 fffff2d1015a4040 __alloc_pages_nodemask systemd-udevd 271288 5.38 fffff2d10d1b3800 __alloc_pages_nodemask dav1d-tile 283460 2.05 fffff2d10cdb0b80 __alloc_pages_nodemask dav1d-tile 283460 2.80 fffff2d10cdb8840 __alloc_pages_nodemask dav1d-tile 283460 2.83 fffff2d10cdb9200 __alloc_pages_nodemask RemVidParent 283460 2.82 fffff2d10cdba200 __alloc_pages_nodemask dav1d-tile 283460 4.11 fffff2d10cdbcf00 __alloc_pages_nodemask journalctl 286270 1.71 fffff2d10e2668c0 __alloc_pages_nodemask journalctl 286270 1.09 fffff2d10ccfac00 __alloc_pages_nodemask journalctl 286270 1.11 fffff2d10cce3200 __alloc_pages_nodemask journalctl 286270 2.96 fffff2d10cce7380 __alloc_pages_nodemask x-terminal-emu 24851 1.14 fffff2d1086a2a00 __alloc_pages_nodemask journalctl 286270 1.44 fffff2d10ccc41c0 __alloc_pages_nodemask kthreadd 2 1.25 fffff2d110c71000 __alloc_pages_nodemask journalctl 286270 1.05 fffff2d10ccb1200 __alloc_pages_nodemask journalctl 286270 1.65 fffff2d10ccbd700 __alloc_pages_nodemask journalctl 286270 3.08 fffff2d102229000 __alloc_pages_nodemask journalctl 286270 1.88 fffff2d10cbfd0c0 __alloc_pages_nodemask journalctl 286270 1.54 fffff2d109088100 __alloc_pages_nodemask journalctl 286270 1.04 fffff2d105db1800 __alloc_pages_nodemask kthreadd 2 2.57 fffff2d1092603c0 __alloc_pages_nodemask journalctl 286270 3.59 fffff2d109358780 __alloc_pages_nodemask journalctl 286270 1.21 fffff2d10ed83980 __alloc_pages_nodemask journalctl 286270 3.17 fffff2d10cb6be40 __alloc_pages_nodemask journalctl 286270 1.65 fffff2d10cb6de00 __alloc_pages_nodemask journalctl 286270 1.00 fffff2d10cb6fdc0 __alloc_pages_nodemask journalctl 286270 3.97 fffff2d10cb46700 __alloc_pages_nodemask journalctl 286270 1.08 fffff2d10fef7700 __alloc_pages_nodemask journalctl 286270 1.11 fffff2d1106ab840 __alloc_pages_nodemask journalctl 286270 1.16 fffff2d10cb204c0 __alloc_pages_nodemask journalctl 286270 1.22 fffff2d110d71640 __alloc_pages_nodemask journalctl 286270 1.10 fffff2d110e4afc0 __alloc_pages_nodemask journalctl 286270 1.03 fffff2d10d912580 __alloc_pages_nodemask journalctl 286270 1.06 fffff2d109de5a00 __alloc_pages_nodemask journalctl 286270 1.05 fffff2d10cad6f40 __alloc_pages_nodemask journalctl 286270 1.52 fffff2d10ca206c0 __alloc_pages_nodemask journalctl 286270 1.17 fffff2d10927f300 __alloc_pages_nodemask journalctl 286270 1.18 fffff2d10ca1a400 __alloc_pages_nodemask kthreadd 2 1.77 fffff2d11008d400 __alloc_pages_nodemask journalctl 286270 2.98 fffff2d108f41e40 __alloc_pages_nodemask journalctl 286270 1.09 fffff2d10c9fe440 __alloc_pages_nodemask journalctl 286270 1.53 fffff2d1090532c0 __alloc_pages_nodemask journalctl 286270 1.45 fffff2d10c9d4e40 __alloc_pages_nodemask kthreadd 2 1.72 fffff2d10fc46040 __alloc_pages_nodemask z_wr_iss_h 684 1.85 fffff2d10a384cc0 __alloc_pages_nodemask journalctl 286270 1.29 fffff2d10c9c0480 __alloc_pages_nodemask kthreadd 2 1.35 fffff2d109ffd8c0 __alloc_pages_nodemask systemd-udevd 288492 2.55 fffff2d10e30a5c0 __alloc_pages_nodemask This feels better but not yet fixed. Thanks -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1861359 Title: swap storms kills interactive use Status in linux package in Ubuntu: Confirmed Status in linux source package in Focal: Confirmed Bug description: [Impact] High watermark boosting can cause large swap activity under certain memory intensive workloads, making the system very unresponsive (screen does not refresh, keyboard not responding, etc.). This large swap activity seems to be prevented disabling high watermark boosting. [Test case] Opening this web page in chrome seems to be a good reproducer of the problem: https://platform.leolabs.space/visualizations/conjunction?type=conjunction&reportId=2004981040 When this page is opened we can clearly see from 'top' (for example) that the used swap is going up very quickly. With the fix applied swap is not used at all and the system is always responsive. [Fix] Set vm.watermark_boost_factor to 0, disabling watermark boosting by default. [Regression potential] Regression potential is minimal, setting vm.watermark_boost_factor to 0 by default restores the old kernel behavior before watermark boosting was introduced. In case of unexpected regressions we can always fix this in user-space via sysctl. [Original report] Hello, several times since upgrading to focal from 19.04 I've found my computer entirely unresponsive for periods of twenty or thirty seconds. No mouse movement, no keyboard input, the screen output does not change. My computer was using swap space and despite very slow writeout speeds well below what the NVME drive can handle, the computer was unusable. I've captured some vmstat 1 output and top output that I started collecting during the event. (Normally one very long painful period is followed by several shorter periods of uselessness.) Thanks ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: linux-image-5.4.0-12-generic 5.4.0-12.15 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 Uname: Linux 5.4.0-12-generic x86_64 NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair ApportVersion: 2.20.11-0ubuntu15 Architecture: amd64 Date: Wed Jan 29 23:44:05 2020 ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: linux-signed-5.4 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) --- ProblemType: Bug AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.4.0-12-generic. ApportVersion: 2.20.11-0ubuntu16 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/controlC0: sarnold 2734 F.... pulseaudio /dev/snd/controlC1: sarnold 2734 F.... pulseaudio Card0.Amixer.info: Card hw:0 'PCH'/'HDA Intel PCH at 0x2fe1028000 irq 145' Mixer name : 'Realtek ALC285' Components : 'HDA:10ec0285,17aa225c,00100002 HDA:8086280b,80860101,00100000' Controls : 53 Simple ctrls : 15 Card1.Amixer.info: Card hw:1 'Audio'/'Generic ThinkPad Dock USB Audio at usb-0000:00:14.0-4.2.4, high speed' Mixer name : 'USB Mixer' Components : 'USB17ef:306f' Controls : 9 Simple ctrls : 4 DistroRelease: Ubuntu 20.04 HibernationDevice: RESUME=none IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: LENOVO 20KHCTO1WW NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu@/vmlinuz-5.4.0-12-generic root=ZFS=rpool/ROOT/ubuntu ro root=ZFS=rpool/ROOT/ubuntu quiet splash acpi_osi=! "acpi_osi=Windows 2015" vt.handoff=1 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 RelatedPackageVersions: linux-restricted-modules-5.4.0-12-generic N/A linux-backports-modules-5.4.0-12-generic N/A linux-firmware 1.185 Tags: focal Uname: Linux 5.4.0-12-generic x86_64 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) UserGroups: adm cdrom libvirt lpadmin plugdev sambashare sbuild sudo _MarkForUpload: True dmi.bios.date: 11/25/2019 dmi.bios.vendor: LENOVO dmi.bios.version: N23ET69W (1.44 ) dmi.board.asset.tag: Not Available dmi.board.name: 20KHCTO1WW dmi.board.vendor: LENOVO dmi.board.version: SDK0J40709 WIN dmi.chassis.asset.tag: No Asset Information dmi.chassis.type: 10 dmi.chassis.vendor: LENOVO dmi.chassis.version: None dmi.modalias: dmi:bvnLENOVO:bvrN23ET69W(1.44):bd11/25/2019:svnLENOVO:pn20KHCTO1WW:pvrThinkPadX1Carbon6th:rvnLENOVO:rn20KHCTO1WW:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrNone: dmi.product.family: ThinkPad X1 Carbon 6th dmi.product.name: 20KHCTO1WW dmi.product.sku: LENOVO_MT_20KH_BU_Think_FM_ThinkPad X1 Carbon 6th dmi.product.version: ThinkPad X1 Carbon 6th dmi.sys.vendor: LENOVO --- ProblemType: Bug AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.4.0-12-generic. ApportVersion: 2.20.11-0ubuntu16 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/controlC0: sarnold 2734 F.... pulseaudio /dev/snd/controlC1: sarnold 2734 F.... pulseaudio Card0.Amixer.info: Card hw:0 'PCH'/'HDA Intel PCH at 0x2fe1028000 irq 145' Mixer name : 'Realtek ALC285' Components : 'HDA:10ec0285,17aa225c,00100002 HDA:8086280b,80860101,00100000' Controls : 53 Simple ctrls : 15 Card1.Amixer.info: Card hw:1 'Audio'/'Generic ThinkPad Dock USB Audio at usb-0000:00:14.0-4.2.4, high speed' Mixer name : 'USB Mixer' Components : 'USB17ef:306f' Controls : 9 Simple ctrls : 4 DistroRelease: Ubuntu 20.04 HibernationDevice: RESUME=none IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' MachineType: LENOVO 20KHCTO1WW NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) ProcEnviron: TERM=rxvt-unicode-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 i915drmfb ProcKernelCmdLine: BOOT_IMAGE=/BOOT/ubuntu@/vmlinuz-5.4.0-12-generic root=ZFS=rpool/ROOT/ubuntu ro root=ZFS=rpool/ROOT/ubuntu quiet splash acpi_osi=! "acpi_osi=Windows 2015" vt.handoff=1 ProcVersionSignature: Ubuntu 5.4.0-12.15-generic 5.4.8 RelatedPackageVersions: linux-restricted-modules-5.4.0-12-generic N/A linux-backports-modules-5.4.0-12-generic N/A linux-firmware 1.185 Tags: focal Uname: Linux 5.4.0-12-generic x86_64 UpgradeStatus: Upgraded to focal on 2020-01-24 (5 days ago) UserGroups: adm cdrom libvirt lpadmin plugdev sambashare sbuild sudo _MarkForUpload: True dmi.bios.date: 11/25/2019 dmi.bios.vendor: LENOVO dmi.bios.version: N23ET69W (1.44 ) dmi.board.asset.tag: Not Available dmi.board.name: 20KHCTO1WW dmi.board.vendor: LENOVO dmi.board.version: SDK0J40709 WIN dmi.chassis.asset.tag: No Asset Information dmi.chassis.type: 10 dmi.chassis.vendor: LENOVO dmi.chassis.version: None dmi.modalias: dmi:bvnLENOVO:bvrN23ET69W(1.44):bd11/25/2019:svnLENOVO:pn20KHCTO1WW:pvrThinkPadX1Carbon6th:rvnLENOVO:rn20KHCTO1WW:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrNone: dmi.product.family: ThinkPad X1 Carbon 6th dmi.product.name: 20KHCTO1WW dmi.product.sku: LENOVO_MT_20KH_BU_Think_FM_ThinkPad X1 Carbon 6th dmi.product.version: ThinkPad X1 Carbon 6th dmi.sys.vendor: LENOVO To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1861359/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
