Hi Rafael, For starters, the server Chris mentioned above didn't panic because the kernel.softlockup_panic wasn't set to 1 on reboot. This is now fixed.
Then, we're still running 3.19 (all the nodes got rebooted to 3.19.0-33-generic). Let me know if you wish us to get back to 3.13. I verified that all the firmwares were the most recent ones, and they were. I rebooted all the nodes with the proper x2apic kernel options. I also disabled all C-States, and also set everything relevant to "performance". You can see the changes here : http://paste.ubuntu.com/13312776/ (this paste is showing all possible settings in G7 and Gen8, I of course could only apply the settings that existed on each infrastructure). Unfortunately, even with all this, we had a G7 that panic'ed and crashdump'ed about ~1h after I set it back in the compute pool. You will find the apport and crashdump below. Let me know what are the next steps. Thanks ! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1505564 Title: Soft lockup with "block nbdX: Attempted send on closed socket" spam Status in linux package in Ubuntu: In Progress Bug description: Hi, Some of our nova compute hosts regularly freeze, sometimes for a few hours, with kern.log getting spammed with : block nbdX: Attempted send on closed socket and a few "CPU soft lockup" messages (see attached log). This clears up when the queue gets cleared, eg : block nbdX: queue cleared trusty hosts with kernel version 3.19.0-30-generic. Note that timestamps from kern.log appears to be wrong, it looks like the messages are being held, and then all delivered at once when the kernel unfreezes. Attaching apport files from 2 hosts below. --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 9 13:14 seq crw-rw---- 1 root audio 116, 33 Oct 9 13:14 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=1bd039df-c419-4cb7-b1ad-fe004d55ccd4 ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 12/08/2012 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd12/08/2012:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 9 12:37 seq crw-rw---- 1 root audio 116, 33 Oct 9 12:37 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.15 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-30-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash ProcVersionSignature: Ubuntu 3.19.0-30.34~14.04.1-generic 3.19.8-ckt6 RelatedPackageVersions: linux-restricted-modules-3.19.0-30-generic N/A linux-backports-modules-3.19.0-30-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-30-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 28 13:07 seq crw-rw---- 1 root audio 116, 33 Oct 28 13:07 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.16 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2 RelatedPackageVersions: linux-restricted-modules-3.13.0-29-generic N/A linux-backports-modules-3.13.0-29-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.13.0-29-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 28 16:31 seq crw-rw---- 1 root audio 116, 33 Oct 28 16:31 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.18 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL385 G7 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 radeondrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-24-generic root=UUID=29988d72-90fe-4329-a0c1-0e3bfb88beab ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.13.0-24.47-generic 3.13.9 RelatedPackageVersions: linux-restricted-modules-3.13.0-24-generic N/A linux-backports-modules-3.13.0-24-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.13.0-24-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: Oct 29 07:58:52 druk kernel: [55713.346292] nr_pdflush_threads exported in /proc is scheduled for removal Oct 29 07:58:52 druk kernel: [55713.346523] sysctl: The scan_unevictable_pages sysctl/node-interface has been disabled for lack of a legitimate use case. If you have one, please send an email to linux...@kvack.org. _MarkForUpload: True dmi.bios.date: 02/02/2014 dmi.bios.vendor: HP dmi.bios.version: A18 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrA18:bd02/02/2014:svnHP:pnProLiantDL385G7:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL385 G7 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 28 19:56 seq crw-rw---- 1 root audio 116, 33 Oct 28 19:56 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.18 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-29-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2 RelatedPackageVersions: linux-restricted-modules-3.13.0-29-generic N/A linux-backports-modules-3.13.0-29-generic N/A linux-firmware 1.127.15 RfKill: Error: [Errno 2] No such file or directory Tags: trusty uec-images Uname: Linux 3.13.0-29-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: Oct 29 07:41:34 orlo kernel: [42337.380135] qbrce3298b1-cb: port 2(tapce3298b1-cb) entered disabled state Oct 29 07:41:34 orlo kernel: [42337.380543] device tapce3298b1-cb left promiscuous mode Oct 29 07:41:34 orlo kernel: [42337.380580] qbrce3298b1-cb: port 2(tapce3298b1-cb) entered disabled state Oct 29 07:41:35 orlo kernel: [42338.223036] type=1400 audit(1446104495.424:137): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="libvirt-80a3c754-9f62-4011-baea-b3a8f37d3746" pid=11554 comm="apparmor_parser" Oct 29 07:41:35 orlo kernel: [42338.308192] qbrce3298b1-cb: port 1(qvbce3298b1-cb) entered disabled state _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 29 21:10 seq crw-rw---- 1 root audio 116, 33 Oct 29 21:10 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.18 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-31-generic root=UUID=f3c6cae8-09dc-4607-8675-c9123ea9c9fd ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.19.0-31.36~14.04.1-generic 3.19.8-ckt7 RelatedPackageVersions: linux-restricted-modules-3.19.0-31-generic N/A linux-backports-modules-3.19.0-31-generic N/A linux-firmware 1.127.16 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-31-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 30 12:31 seq crw-rw---- 1 root audio 116, 33 Oct 30 12:31 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.18 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 14.04 IwConfig: Error: [Errno 2] No such file or directory MachineType: HP ProLiant DL360p Gen8 Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-31-generic root=UUID=46ac2c1e-5f16-45bd-b383-e952f78fd142 ro console=tty0 console=ttyS1,38400 nosplash crashkernel=384M-:512M ProcVersionSignature: Ubuntu 3.19.0-31.36~14.04.1-generic 3.19.8-ckt7 RelatedPackageVersions: linux-restricted-modules-3.19.0-31-generic N/A linux-backports-modules-3.19.0-31-generic N/A linux-firmware 1.127.16 RfKill: Error: [Errno 2] No such file or directory StagingDrivers: visorchannel visorutil Tags: trusty uec-images staging Uname: Linux 3.19.0-31-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: True dmi.bios.date: 11/14/2013 dmi.bios.vendor: HP dmi.bios.version: P71 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.modalias: dmi:bvnHP:bvrP71:bd11/14/2013:svnHP:pnProLiantDL360pGen8:pvr:cvnHP:ct23:cvr: dmi.product.name: ProLiant DL360p Gen8 dmi.sys.vendor: HP To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1505564/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
