Hi
I am experiencing at one of our compute node hypervisor. kernel version
we are using is 4.4.0-83, but seems to be having the issue reported in
this report.
[Mon Aug 7 00:19:42 2017] nova-compute invoked oom-killer: gfp_mask=0x2c200ca,
order=0, oom_score_adj=0
[Mon Aug 7 00:19:42 2017] nova-compute cpuset=/ mems_allowed=0-1
[Mon Aug 7 00:19:42 2017] CPU: 7 PID: 2164484 Comm: nova-compute Tainted: G
OE 4.4.0-83-generic #106-Ubuntu
[Mon Aug 7 00:19:42 2017] Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS
2.4.3 01/17/2017
[Mon Aug 7 00:19:42 2017] 0000000000000286 00000000d6004dce ffff88014e753a50
ffffffff813f9513
[Mon Aug 7 00:19:42 2017] ffff88014e753c08 ffff883fecf88e00 ffff88014e753ac0
ffffffff8120b53e
[Mon Aug 7 00:19:42 2017] 0000000000000015 0000000000000000 ffff881fe883b740
ffff883fe94f7000
[Mon Aug 7 00:19:42 2017] Call Trace:
[Mon Aug 7 00:19:42 2017] [<ffffffff813f9513>] dump_stack+0x63/0x90
[Mon Aug 7 00:19:42 2017] [<ffffffff81391c64>] ? apparmor_capable+0xc4/0x1b0
[Mon Aug 7 00:19:42 2017] [<ffffffff81192ae2>] oom_kill_process+0x202/0x3c0
[Mon Aug 7 00:19:42 2017] [<ffffffff81192f09>] out_of_memory+0x219/0x460
[Mon Aug 7 00:19:42 2017] [<ffffffff81198ef8>]
__alloc_pages_slowpath.constprop.88+0x938/0xad0
[Mon Aug 7 00:19:42 2017] [<ffffffff81199316>]
__alloc_pages_nodemask+0x286/0x2a0
[Mon Aug 7 00:19:42 2017] [<ffffffff811e467d>] alloc_pages_vma+0xad/0x250
[Mon Aug 7 00:19:42 2017] [<ffffffff811fad53>] do_huge_pmd_wp_page+0x153/0xb70
[Mon Aug 7 00:19:42 2017] [<ffffffff811c1a5f>] handle_mm_fault+0x90f/0x1820
[Mon Aug 7 00:19:42 2017] [<ffffffff8106b802>] ? do_page_fault+0x22/0x30
[Mon Aug 7 00:19:42 2017] [<ffffffff81842cf8>] ? page_fault+0x28/0x30
[Mon Aug 7 00:19:42 2017] [<ffffffff8106b577>] __do_page_fault+0x197/0x400
[Mon Aug 7 00:19:42 2017] [<ffffffff8106b802>] do_page_fault+0x22/0x30
[Mon Aug 7 00:19:42 2017] [<ffffffff81842cf8>] page_fault+0x28/0x30
[Mon Aug 7 00:19:42 2017] Mem-Info:
[Mon Aug 7 00:19:42 2017] active_anon:61350709 inactive_anon:2118817
isolated_anon:0
active_file:0 inactive_file:0 isolated_file:32
unevictable:915 dirty:0 writeback:8 unstable:0
slab_reclaimable:14082 slab_unreclaimable:64456
mapped:3492 shmem:329012 pagetables:142167 bounce:0
free:260204 free_pcp:4111 free_cma:0
[Tue Aug 8 05:50:08 2017] apt-check invoked oom-killer: gfp_mask=0x24201ca,
order=0, oom_score_adj=0
[Tue Aug 8 05:50:08 2017] apt-check cpuset=/ mems_allowed=0-1
[Tue Aug 8 05:50:08 2017] CPU: 11 PID: 2538289 Comm: apt-check Tainted: G
OE 4.4.0-83-generic #106-Ubuntu
[Tue Aug 8 05:50:08 2017] Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS
2.4.3 01/17/2017
[Tue Aug 8 05:50:08 2017] 0000000000000286 000000005e467cc9 ffff8820b44a39f8
ffffffff813f9513
[Tue Aug 8 05:50:08 2017] ffff8820b44a3bb0 ffff881fec15b800 ffff8820b44a3a68
ffffffff8120b53e
[Tue Aug 8 05:50:08 2017] 0000000000000015 ffffffff81e42ac0 ffff883fe996f980
ffffffffffffff04
[Tue Aug 8 05:50:08 2017] Call Trace:
[Tue Aug 8 05:50:08 2017] [<ffffffff813f9513>] dump_stack+0x63/0x90
[Tue Aug 8 05:50:08 2017] [<ffffffff8134544d>] ? cap_capable+0xd/0x70
[Tue Aug 8 05:50:08 2017] [<ffffffff81192ae2>] oom_kill_process+0x202/0x3c0
[Tue Aug 8 05:50:08 2017] [<ffffffff81192f09>] out_of_memory+0x219/0x460
[Tue Aug 8 05:50:08 2017] [<ffffffff81198ef8>]
__alloc_pages_slowpath.constprop.88+0x938/0xad0
[Tue Aug 8 05:50:08 2017] [<ffffffff81199316>]
__alloc_pages_nodemask+0x286/0x2a0
[Tue Aug 8 05:50:08 2017] [<ffffffff811e2e3c>] alloc_pages_current+0x8c/0x110
[Tue Aug 8 05:50:08 2017] [<ffffffff8118f0ab>] __page_cache_alloc+0xab/0xc0
[Tue Aug 8 05:50:08 2017] [<ffffffff811915ba>] filemap_fault+0x14a/0x3f0
[Tue Aug 8 05:50:08 2017] [<ffffffff812a3506>] ext4_filemap_fault+0x36/0x50
[Tue Aug 8 05:50:08 2017] [<ffffffff811be5d0>] __do_fault+0x50/0xe0
[Tue Aug 8 05:50:08 2017] [<ffffffff811c20f2>] handle_mm_fault+0xfa2/0x1820
[Tue Aug 8 05:50:08 2017] [<ffffffff8106b577>] __do_page_fault+0x197/0x400
[Tue Aug 8 05:50:08 2017] [<ffffffff8106b802>] do_page_fault+0x22/0x30
[Tue Aug 8 05:50:08 2017] [<ffffffff81842cf8>] page_fault+0x28/0x30
[Tue Aug 8 05:50:08 2017] Mem-Info:
[Tue Aug 8 05:50:08 2017] active_anon:61377850 inactive_anon:2049156
isolated_anon:0
active_file:0 inactive_file:0 isolated_file:0
unevictable:915 dirty:0 writeback:0 unstable:0
slab_reclaimable:15329 slab_unreclaimable:101408
mapped:3655 shmem:338468 pagetables:141874 bounce:0
free:260450 free_pcp:2714 free_cma:0
Linux rcgpudc1rh31-02 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1655842
Title:
"Out of memory" errors after upgrade to 4.4.0-59
Status in linux package in Ubuntu:
Fix Released
Status in linux-raspi2 package in Ubuntu:
Confirmed
Status in linux source package in Xenial:
Fix Released
Status in linux-raspi2 source package in Xenial:
Confirmed
Bug description:
I recently replaced some Xenial servers, and started experiencing "Out
of memory" problems with the default kernel.
We bake Amazon AMIs based on an official Ubuntu-provided image (ami-
e6b58e85, in ap-southeast-2, from https://cloud-
images.ubuntu.com/locator/ec2/). Previous versions of our AMI
included "4.4.0-57-generic", but the latest version picked up
"4.4.0-59-generic" as part of a "dist-upgrade".
Instances booted using the new AMI have been using more memory, and
experiencing OOM issues - sometimes during boot, and sometimes a while
afterwards. An example from the system log is:
[ 130.113411] cloud-init[1560]: Cloud-init v. 0.7.8 running 'modules:final'
at Wed, 11 Jan 2017 22:07:53 +0000. Up 29.28 seconds.
[ 130.124219] cloud-init[1560]: Cloud-init v. 0.7.8 finished at Wed, 11 Jan
2017 22:09:35 +0000. Datasource DataSourceEc2. Up 130.09 seconds
[29871.137128] Out of memory: Kill process 2920 (ruby) score 107 or sacrifice
child
[29871.140816] Killed process 2920 (ruby) total-vm:675048kB,
anon-rss:51184kB, file-rss:2164kB
[29871.449209] Out of memory: Kill process 3257 (splunkd) score 97 or
sacrifice child
[29871.453282] Killed process 3258 (splunkd) total-vm:66272kB,
anon-rss:6676kB, file-rss:0kB
[29871.677910] Out of memory: Kill process 2647 (fluentd) score 51 or
sacrifice child
[29871.681872] Killed process 2647 (fluentd) total-vm:117944kB,
anon-rss:23956kB, file-rss:1356kB
I have a hunch that this may be related to the fix for
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1647400,
introduced in linux (4.4.0-58.79).
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-59-generic 4.4.0-59.80
ProcVersionSignature: User Name 4.4.0-59.80-generic 4.4.35
Uname: Linux 4.4.0-59-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jan 12 06:29 seq
crw-rw---- 1 root audio 116, 33 Jan 12 06:29 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.4
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq',
'/dev/snd/timer'] failed with exit code 1:
Date: Thu Jan 12 06:38:45 2017
Ec2AMI: ami-0f93966c
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: ap-southeast-2a
Ec2InstanceType: t2.nano
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Xen HVM domU
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 cirrusdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-59-generic
root=UUID=fb0fef08-f3c5-40bf-9776-f7ba00fe72be ro console=tty1 console=ttyS0
RelatedPackageVersions:
linux-restricted-modules-4.4.0-59-generic N/A
linux-backports-modules-4.4.0-59-generic N/A
linux-firmware 1.157.6
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/09/2016
dmi.bios.vendor: Xen
dmi.bios.version: 4.2.amazon
dmi.chassis.type: 1
dmi.chassis.vendor: Xen
dmi.modalias:
dmi:bvnXen:bvr4.2.amazon:bd12/09/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
dmi.product.name: HVM domU
dmi.product.version: 4.2.amazon
dmi.sys.vendor: Xen
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
