After quite a bit of experimentation I found that I can reproduce the bug if I have zram *and* also swap on the filesystem enabled while exercising the brk stressors and aiol (to cause lots of I/O). Eventually the system grinds to a halt, we lose interactivity and we eventually get lockups as follows: [ 2012.040006] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [stress-ng-brk:1632] [ 2012.040922] Modules linked in: zram(E) kvm_intel(E) kvm(E) irqbypass(E) crct10dif_pclmul(E) crc32_pclmul(E) ghash_clmulni_intel(E) pcbc(E) aesni_intel(E) aes_x86_64(E) crypto_simd(E) glue_helper(E) cryptd(E) psmouse(E) input_leds(E) floppy(E) virtio_scsi(E) serio_raw(E) i2c_piix4(E) mac_hid(E) pata_acpi(E) qemu_fw_cfg(E) 9pnet_virtio(E) 9p(E) 9pnet(E) fscache(E) [ 2012.044655] CPU: 2 PID: 1632 Comm: stress-ng-brk Tainted: G EL 4.15.18 #1 [ 2012.045581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.13.0-1 04/01/2014 [ 2012.046555] RIP: 0010:__raw_callee_save___pv_queued_spin_unlock+0x10/0x17 [ 2012.047340] RSP: 0018:ffffb73382083718 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11 [ 2012.048238] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000002 [ 2012.049078] RDX: 0000000000000000 RSI: ffff9d327c2f6918 RDI: ffffffffa3269978 [ 2012.049909] RBP: ffffb73382083720 R08: ffff9d327c2f6918 R09: ffff9d327c0a5328 [ 2012.050746] R10: ffff9d327c1e2310 R11: ffff9d327c1e2328 R12: ffff9d327c2f6800 [ 2012.051574] R13: ffff9d327c1e2328 R14: ffff9d327c1e2310 R15: ffff9d327c1e2200 [ 2012.052436] FS: 00007f89f2ccd740(0000) GS:ffff9d327f280000(0000) knlGS:0000000000000000 [ 2012.053382] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 2012.054058] CR2: 00007f1350a8dd90 CR3: 00000000311a4004 CR4: 0000000000160ee0 [ 2012.054889] Call Trace: [ 2012.055192] get_swap_pages+0x193/0x360 [ 2012.055652] get_swap_page+0x13f/0x1e0 [ 2012.056123] add_to_swap+0x14/0x70 [ 2012.056530] shrink_page_list+0x81d/0xbc0 [ 2012.057013] shrink_inactive_list+0x242/0x590 [ 2012.057523] shrink_node_memcg+0x364/0x770 [ 2012.058012] shrink_node+0xf7/0x300 [ 2012.058432] ? shrink_node+0xf7/0x300 [ 2012.058863] do_try_to_free_pages+0xc9/0x330 [ 2012.059368] try_to_free_pages+0xee/0x1b0 [ 2012.059842] __alloc_pages_slowpath+0x3fc/0xe00 [ 2012.060424] __alloc_pages_nodemask+0x29a/0x2c0 [ 2012.060963] alloc_pages_vma+0x88/0x1f0 [ 2012.061414] __handle_mm_fault+0x8b7/0x12e0 [ 2012.061909] handle_mm_fault+0xb1/0x210 [ 2012.062375] __do_page_fault+0x281/0x4b0 [ 2012.062848] do_page_fault+0x2e/0xe0 [ 2012.063274] ? async_page_fault+0x2f/0x50 [ 2012.063751] do_async_page_fault+0x51/0x80 [ 2012.064262] async_page_fault+0x45/0x50 [ 2012.064719] RIP: 0033:0x55ec1997bd0a [ 2012.065147] RSP: 002b:00007ffeacd21600 EFLAGS: 00010246 [ 2012.065754] RAX: 000055ec28601000 RBX: 0000000000000005 RCX: 00007f89f2de956b [ 2012.066580] RDX: 000055ec28601000 RSI: 00007ffeacd216d0 RDI: 000055ec28602000 [ 2012.067410] RBP: 00007ffeacd216c0 R08: 0000000000000000 R09: 00007f89f3d0c2f0 [ 2012.068290] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 2012.069129] R13: 0000000000000002 R14: 0000000000000001 R15: 00007ffeacd216d0 [ 2012.069965] Code: 50 41 51 41 52 41 53 e8 3b 05 00 00 41 5b 41 5a 41 59 41 58 5f 5e 5a 59 5d c3 90 55 48 89 e5 52 b8 01 00 00 00 31 d2 f0 0f b0 17 <3c> 01 75 03 5a 5d c3 56 0f b6 f0 e8 bc ff ff ff 5e 5a 5d c3 0f
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1799497 Title: 4.15 kernel hard lockup about once a week Status in linux package in Ubuntu: Incomplete Status in zram-config package in Ubuntu: Incomplete Status in linux source package in Bionic: Confirmed Status in zram-config source package in Bionic: Confirmed Bug description: My main server has been running into hard lockups about once a week ever since I switched to the 4.15 Ubuntu 18.04 kernel. When this happens, nothing is printed to the console, it's effectively stuck showing a login prompt. The system is running with panic=1 on the cmdline but isn't rebooting so the kernel isn't even processing this as a kernel panic. As this felt like a potential hardware issue, I had my hosting provider give me a completely different system, different motherboard, different CPU, different RAM and different storage, I installed that system on 18.04 and moved my data over, a week later, I hit the issue again. We've since also had a LXD user reporting similar symptoms here also on varying hardware: https://github.com/lxc/lxd/issues/5197 My system doesn't have a lot of memory pressure with about 50% of free memory: root@vorash:~# free -m total used free shared buff/cache available Mem: 31819 17574 402 513 13842 13292 Swap: 15909 2687 13222 I will now try to increase console logging as much as possible on the system in the hopes that next time it hangs we can get a better idea of what happened but I'm not too hopeful given the complete silence on the console when this occurs. System is currently on: Linux vorash 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux But I've seen this since the GA kernel on 4.15 so it's not a recent regression. --- ProblemType: Bug AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 23 16:12 seq crw-rw---- 1 root audio 116, 33 Oct 23 16:12 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Cannot stat file /proc/22822/fd/10: Permission denied Cannot stat file /proc/22831/fd/10: Permission denied DistroRelease: Ubuntu 18.04 HibernationDevice: RESUME=none CRYPTSETUP=n IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' Lsusb: Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 002: ID 046b:ff10 American Megatrends, Inc. Virtual Keyboard and Mouse Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Intel Corporation S1200SP NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 mgadrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-38-generic root=UUID=575c878a-0be6-4806-9c83-28f67aedea65 ro biosdevname=0 net.ifnames=0 panic=1 verbose console=tty0 console=ttyS0,115200n8 ProcVersionSignature: Ubuntu 4.15.0-38.41-generic 4.15.18 RelatedPackageVersions: linux-restricted-modules-4.15.0-38-generic N/A linux-backports-modules-4.15.0-38-generic N/A linux-firmware 1.173.1 RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' Tags: bionic Uname: Linux 4.15.0-38-generic x86_64 UnreportableReason: This report is about a package that is not installed. UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: False dmi.bios.date: 01/25/2018 dmi.bios.vendor: Intel Corporation dmi.bios.version: S1200SP.86B.03.01.1029.012520180838 dmi.board.asset.tag: Base Board Asset Tag dmi.board.name: S1200SP dmi.board.vendor: Intel Corporation dmi.board.version: H57532-271 dmi.chassis.asset.tag: .................... dmi.chassis.type: 23 dmi.chassis.vendor: ............................... dmi.chassis.version: .................. dmi.modalias: dmi:bvnIntelCorporation:bvrS1200SP.86B.03.01.1029.012520180838:bd01/25/2018:svnIntelCorporation:pnS1200SP:pvr....................:rvnIntelCorporation:rnS1200SP:rvrH57532-271:cvn...............................:ct23:cvr..................: dmi.product.family: Family dmi.product.name: S1200SP dmi.product.version: .................... dmi.sys.vendor: Intel Corporation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1799497/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
