apport information ** Tags added: apport-collected wily
** Description changed: I am experiencing a memory management problem with 15.10 that I did not experience with 15.04. I have a 24-core (48 thread) server with 64G of RAM. I am getting some strange behavior with respect to swapping and physical memory use. You can see my problem in 'top': top - 18:52:09 up 1 day, 2:25, 3 users, load average: 1.64, 1.30, 1.18 Tasks: 525 total, 2 running, 523 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.3 us, 1.3 sy, 0.0 ni, 97.0 id, 1.4 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 65937528 total, 37526160 used, 28411368 free, 14396 buffers KiB Swap: 67071996 total, 67071724 used, 272 free. 104304 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 363 root 20 0 0 0 0 R 100.0 0.0 324:02.98 kswapd0 6725 theosib 20 0 99.398g 0.034t 8920 D 12.0 55.1 59:08.71 common_shell_ex There's a single user logged in (me), and I have a single process using a large amount of virtual memory. However, something is limiting it to around half the physical memory, while the swap partition is basically full. I've watched these processes (Synopsys Design Compiler) run, and they don't break the 50% mark until swap fills. And another weird thing is that kswapd0 uses very high (usually 100%) CPU. AFAIK, kswapd0 should be I/O bound and therefore not use a lot of CPU time. I've looked to see if there were any limits being imposed, but ulimit says otherwise: $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 257447 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 257447 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited I've also tried setting swappiness to 10, but that didn't help. This is a pretty serious problem. What used to take a few hours on 15.04 now takes more than 5 to 10 times longer, because the process is forced to wait on swap, which is reading and writing at about 20M/sec each. So it's hammering my SSD that contains the swap partition and going really slow. - I've done some googling about kswapd0 using high CPU, and all I can work - out is that some people think it's a kernel bug. + I've done some googling about kswapd0 using high CPU, and all I can work out is that some people think it's a kernel bug. + --- + AlsaDevices: + total 0 + crw-rw---- 1 root audio 116, 1 Nov 6 13:28 seq + crw-rw---- 1 root audio 116, 33 Nov 6 13:28 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.19.1-0ubuntu3 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + DistroRelease: Ubuntu 15.10 + HibernationDevice: RESUME=UUID=e577be79-e4a4-41d9-b946-cd0cd50d05ba + InstallationDate: Installed on 2013-11-27 (709 days ago) + InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Release amd64 (20120424.1) + MachineType: Supermicro X9DRW + Package: linux (not installed) + PciMultimedia: + + ProcEnviron: + TERM=screen + PATH=(custom, no user) + XDG_RUNTIME_DIR=<set> + LANG=en_US.UTF-8 + SHELL=/bin/bash + ProcFB: 0 VESA VGA + ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-16-generic root=UUID=08710eb9-367c-4be0-ba25-700476f690fc ro + ProcVersionSignature: Ubuntu 4.2.0-16.19-generic 4.2.3 + RelatedPackageVersions: + linux-restricted-modules-4.2.0-16-generic N/A + linux-backports-modules-4.2.0-16-generic N/A + linux-firmware 1.149 + RfKill: Error: [Errno 2] No such file or directory + Tags: wily + Uname: Linux 4.2.0-16-generic x86_64 + UpgradeStatus: Upgraded to wily on 2015-10-26 (11 days ago) + UserGroups: adm audio cdrom dip kvm lpadmin plugdev sambashare sudo video + _MarkForUpload: True + dmi.bios.date: 08/08/2013 + dmi.bios.vendor: American Megatrends Inc. + dmi.bios.version: 3.0a + dmi.board.asset.tag: To be filled by O.E.M. + dmi.board.name: X9DRW + dmi.board.vendor: Supermicro + dmi.board.version: 0123456789 + dmi.chassis.asset.tag: To Be Filled By O.E.M. + dmi.chassis.type: 3 + dmi.chassis.vendor: Supermicro + dmi.chassis.version: 0123456789 + dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3.0a:bd08/08/2013:svnSupermicro:pnX9DRW:pvr0123456789:rvnSupermicro:rnX9DRW:rvr0123456789:cvnSupermicro:ct3:cvr0123456789: + dmi.product.name: X9DRW + dmi.product.version: 0123456789 + dmi.sys.vendor: Supermicro ** Attachment added: "CRDA.txt" https://bugs.launchpad.net/bugs/1513673/+attachment/4514730/+files/CRDA.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1513673 Title: 15.10 swapping heavily after process exceeds 50% physical RAM Status in linux package in Ubuntu: Incomplete Bug description: I am experiencing a memory management problem with 15.10 that I did not experience with 15.04. I have a 24-core (48 thread) server with 64G of RAM. I am getting some strange behavior with respect to swapping and physical memory use. You can see my problem in 'top': top - 18:52:09 up 1 day, 2:25, 3 users, load average: 1.64, 1.30, 1.18 Tasks: 525 total, 2 running, 523 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.3 us, 1.3 sy, 0.0 ni, 97.0 id, 1.4 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem: 65937528 total, 37526160 used, 28411368 free, 14396 buffers KiB Swap: 67071996 total, 67071724 used, 272 free. 104304 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 363 root 20 0 0 0 0 R 100.0 0.0 324:02.98 kswapd0 6725 theosib 20 0 99.398g 0.034t 8920 D 12.0 55.1 59:08.71 common_shell_ex There's a single user logged in (me), and I have a single process using a large amount of virtual memory. However, something is limiting it to around half the physical memory, while the swap partition is basically full. I've watched these processes (Synopsys Design Compiler) run, and they don't break the 50% mark until swap fills. And another weird thing is that kswapd0 uses very high (usually 100%) CPU. AFAIK, kswapd0 should be I/O bound and therefore not use a lot of CPU time. I've looked to see if there were any limits being imposed, but ulimit says otherwise: $ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 257447 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 257447 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited I've also tried setting swappiness to 10, but that didn't help. This is a pretty serious problem. What used to take a few hours on 15.04 now takes more than 5 to 10 times longer, because the process is forced to wait on swap, which is reading and writing at about 20M/sec each. So it's hammering my SSD that contains the swap partition and going really slow. I've done some googling about kswapd0 using high CPU, and all I can work out is that some people think it's a kernel bug. --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Nov 6 13:28 seq crw-rw---- 1 root audio 116, 33 Nov 6 13:28 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.19.1-0ubuntu3 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: DistroRelease: Ubuntu 15.10 HibernationDevice: RESUME=UUID=e577be79-e4a4-41d9-b946-cd0cd50d05ba InstallationDate: Installed on 2013-11-27 (709 days ago) InstallationMedia: Ubuntu-Server 12.04 LTS "Precise Pangolin" - Release amd64 (20120424.1) MachineType: Supermicro X9DRW Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-16-generic root=UUID=08710eb9-367c-4be0-ba25-700476f690fc ro ProcVersionSignature: Ubuntu 4.2.0-16.19-generic 4.2.3 RelatedPackageVersions: linux-restricted-modules-4.2.0-16-generic N/A linux-backports-modules-4.2.0-16-generic N/A linux-firmware 1.149 RfKill: Error: [Errno 2] No such file or directory Tags: wily Uname: Linux 4.2.0-16-generic x86_64 UpgradeStatus: Upgraded to wily on 2015-10-26 (11 days ago) UserGroups: adm audio cdrom dip kvm lpadmin plugdev sambashare sudo video _MarkForUpload: True dmi.bios.date: 08/08/2013 dmi.bios.vendor: American Megatrends Inc. dmi.bios.version: 3.0a dmi.board.asset.tag: To be filled by O.E.M. dmi.board.name: X9DRW dmi.board.vendor: Supermicro dmi.board.version: 0123456789 dmi.chassis.asset.tag: To Be Filled By O.E.M. dmi.chassis.type: 3 dmi.chassis.vendor: Supermicro dmi.chassis.version: 0123456789 dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3.0a:bd08/08/2013:svnSupermicro:pnX9DRW:pvr0123456789:rvnSupermicro:rnX9DRW:rvr0123456789:cvnSupermicro:ct3:cvr0123456789: dmi.product.name: X9DRW dmi.product.version: 0123456789 dmi.sys.vendor: Supermicro To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1513673/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
