Additional logs from 4.4.0 machine by running `dmesg -w` until crash.

[ 3031.276097] ------------[ cut here ]------------
[ 3031.276106] WARNING: CPU: 0 PID: 31804 at 
/build/linux-0uniEn/linux-4.4.0/net/ipv6/addrconf_core.c:159 
in6_dev_finish_destroy+0x6b/0xc0()
[ 3031.276108] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp 
ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 
nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack 
x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs isofs ppdev 
input_leds serio_raw
 parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul crc32_pclmul 
ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper 
cryptd psmouse floppy
[ 3031.276131] CPU: 0 PID: 31804 Comm: kworker/u30:3 Not tainted 
4.4.0-83-generic #106-Ubuntu
[ 3031.276132] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
[ 3031.276136] Workqueue: netns cleanup_net
[ 3031.276137]  0000000000000286 00000000e9d24b1b ffff880043383ba0 
ffffffff813f9513
[ 3031.276140]  0000000000000000 ffffffff81d75940 ffff880043383bd8 
ffffffff81081322
[ 3031.276141]  ffff880070297800 0000000000000000 0000000000000006 
ffff880043383ca8
[ 3031.276143] Call Trace:
[ 3031.276148]  [<ffffffff813f9513>] dump_stack+0x63/0x90
[ 3031.276151]  [<ffffffff81081322>] warn_slowpath_common+0x82/0xc0
[ 3031.276153]  [<ffffffff8108146a>] warn_slowpath_null+0x1a/0x20
[ 3031.276154]  [<ffffffff8181ba0b>] in6_dev_finish_destroy+0x6b/0xc0
[ 3031.276157]  [<ffffffff817f17e6>] ip6_route_dev_notify+0x116/0x130
[ 3031.276159]  [<ffffffff810a1b1a>] notifier_call_chain+0x4a/0x70
[ 3031.276161]  [<ffffffff810a1c96>] raw_notifier_call_chain+0x16/0x20
[ 3031.276163]  [<ffffffff8172f655>] call_netdevice_notifiers_info+0x35/0x60
[ 3031.276165]  [<ffffffff81739e7d>] netdev_run_todo+0x16d/0x320
[ 3031.276168]  [<ffffffff817316c9>] ? rollback_registered_many+0x2c9/0x340
[ 3031.276171]  [<ffffffff8174540e>] rtnl_unlock+0xe/0x10
[ 3031.276173]  [<ffffffff81732a37>] default_device_exit_batch+0x147/0x170
[ 3031.276176]  [<ffffffff810c4140>] ? __wake_up_sync+0x20/0x20
[ 3031.276178]  [<ffffffff8172a892>] ops_exit_list.isra.4+0x52/0x60
[ 3031.276179]  [<ffffffff8172b8e2>] cleanup_net+0x1c2/0x2a0
[ 3031.276182]  [<ffffffff8109a585>] process_one_work+0x165/0x480
[ 3031.276184]  [<ffffffff8109a8eb>] worker_thread+0x4b/0x4c0
[ 3031.276186]  [<ffffffff8109a8a0>] ? process_one_work+0x480/0x480
[ 3031.276187]  [<ffffffff810a0c25>] kthread+0xe5/0x100
[ 3031.276189]  [<ffffffff810a0b40>] ? kthread_create_on_node+0x1e0/0x1e0
[ 3031.276192]  [<ffffffff81840f0f>] ret_from_fork+0x3f/0x70
[ 3031.276193]  [<ffffffff810a0b40>] ? kthread_create_on_node+0x1e0/0x1e0
[ 3031.276194] ---[ end trace 5d921dfe814c9c7c ]---
[ 3031.276195] ------------[ cut here ]------------
[ 3031.276197] WARNING: CPU: 0 PID: 31804 at 
/build/linux-0uniEn/linux-4.4.0/net/ipv6/addrconf_core.c:160 
in6_dev_finish_destroy+0x83/0xc0()
[ 3031.276198] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp 
ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 
nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack 
x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs isofs ppdev 
input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq 
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw 
gf128mul glue_helper ablk_helper cryptd psmouse floppy
[ 3031.276211] CPU: 0 PID: 31804 Comm: kworker/u30:3 Tainted: G        W       
4.4.0-83-generic #106-Ubuntu
[ 3031.276212] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
[ 3031.276213] Workqueue: netns cleanup_net
[ 3031.276214]  0000000000000286 00000000e9d24b1b ffff880043383ba0 
ffffffff813f9513
[ 3031.276216]  0000000000000000 ffffffff81d75940 ffff880043383bd8 
ffffffff81081322
[ 3031.276217]  ffff880070297800 0000000000000000 0000000000000006 
ffff880043383ca8
[ 3031.276219] Call Trace:
[ 3031.276220]  [<ffffffff813f9513>] dump_stack+0x63/0x90
[ 3031.276222]  [<ffffffff81081322>] warn_slowpath_common+0x82/0xc0
[ 3031.276224]  [<ffffffff8108146a>] warn_slowpath_null+0x1a/0x20
[ 3031.276225]  [<ffffffff8181ba23>] in6_dev_finish_destroy+0x83/0xc0
[ 3031.276226]  [<ffffffff817f17e6>] ip6_route_dev_notify+0x116/0x130
[ 3031.276228]  [<ffffffff810a1b1a>] notifier_call_chain+0x4a/0x70
[ 3031.276229]  [<ffffffff810a1c96>] raw_notifier_call_chain+0x16/0x20
[ 3031.276231]  [<ffffffff8172f655>] call_netdevice_notifiers_info+0x35/0x60
[ 3031.276233]  [<ffffffff81739e7d>] netdev_run_todo+0x16d/0x320
[ 3031.276234]  [<ffffffff817316c9>] ? rollback_registered_many+0x2c9/0x340
[ 3031.276237]  [<ffffffff8174540e>] rtnl_unlock+0xe/0x10
[ 3031.276239]  [<ffffffff81732a37>] default_device_exit_batch+0x147/0x170
[ 3031.276240]  [<ffffffff810c4140>] ? __wake_up_sync+0x20/0x20
[ 3031.276242]  [<ffffffff8172a892>] ops_exit_list.isra.4+0x52/0x60
[ 3031.276243]  [<ffffffff8172b8e2>] cleanup_net+0x1c2/0x2a0
[ 3031.276245]  [<ffffffff8109a585>] process_one_work+0x165/0x480
[ 3031.276247]  [<ffffffff8109a8eb>] worker_thread+0x4b/0x4c0
[ 3031.276249]  [<ffffffff8109a8a0>] ? process_one_work+0x480/0x480
[ 3031.276250]  [<ffffffff810a0c25>] kthread+0xe5/0x100
[ 3031.276251]  [<ffffffff810a0b40>] ? kthread_create_on_node+0x1e0/0x1e0
[ 3031.276253]  [<ffffffff81840f0f>] ret_from_fork+0x3f/0x70
[ 3031.276254]  [<ffffffff810a0b40>] ? kthread_create_on_node+0x1e0/0x1e0
[ 3031.276255] ---[ end trace 5d921dfe814c9c7d ]---
[ 3031.276260] BUG: unable to handle kernel NULL pointer dereference at 
0000000000000448
[ 3031.280059] IP: [<ffffffff8181b9cd>] in6_dev_finish_destroy+0x2d/0xc0
[ 3031.280059] PGD 7bbf6067 PUD 7ba63067 PMD 0
[ 3031.280059] Oops: 0000 [#1] SMP
[ 3031.280059] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp 
ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 
nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack 
x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs isofs ppdev 
input_leds serio_raw parport_pc parport autofs4 btrfs xor raid6_pq 
crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw 
gf128mul glue_helper ablk_helper cryptd psmouse floppy
[ 3031.280059] CPU: 0 PID: 31804 Comm: kworker/u30:3 Tainted: G        W       
4.4.0-83-generic #106-Ubuntu
[ 3031.280059] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
[ 3031.280059] Workqueue: netns cleanup_net
[ 3031.280059] task: ffff88001608c600 ti: ffff880043380000 task.ti: 
ffff880043380000
[ 3031.280059] RIP: 0010:[<ffffffff8181b9cd>]  [<ffffffff8181b9cd>] 
in6_dev_finish_destroy+0x2d/0xc0
[ 3031.280059] RSP: 0018:ffff880043383bf8  EFLAGS: 00010246
[ 3031.280059] RAX: 0000000000000024 RBX: ffff880070297800 RCX: 0000000000000006
[ 3031.280059] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000009
[ 3031.280059] RBP: ffff880043383c08 R08: 000000000000000a R09: 00000000000002e0
[ 3031.280059] R10: 0000000000000004 R11: 00000000000002e0 R12: 0000000000000000
[ 3031.280059] R13: 0000000000000006 R14: ffff880043383ca8 R15: 0000000000000000
[ 3031.280059] FS:  0000000000000000(0000) GS:ffff88007d200000(0000) 
knlGS:0000000000000000
[ 3031.280059] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3031.280059] CR2: 0000000000000448 CR3: 0000000036d4c000 CR4: 00000000001406f0
[ 3031.280059] Stack:
[ 3031.280059]  ffff880007803100 0000000000000000 ffff880043383c20 
ffffffff817f17e6
[ 3031.280059]  00000000ffffffed ffff880043383c58 ffffffff810a1b1a 
ffff880043383ca8
[ 3031.280059]  0000000000000006 ffff880005580000 00000001000a6343 
00000000000000fa
[ 3031.280059] Call Trace:
[ 3031.280059]  [<ffffffff817f17e6>] ip6_route_dev_notify+0x116/0x130
[ 3031.280059]  [<ffffffff810a1b1a>] notifier_call_chain+0x4a/0x70
[ 3031.280059]  [<ffffffff810a1c96>] raw_notifier_call_chain+0x16/0x20
[ 3031.280059]  [<ffffffff8172f655>] call_netdevice_notifiers_info+0x35/0x60
[ 3031.280059]  [<ffffffff81739e7d>] netdev_run_todo+0x16d/0x320
[ 3031.280059]  [<ffffffff817316c9>] ? rollback_registered_many+0x2c9/0x340
[ 3031.280059]  [<ffffffff8174540e>] rtnl_unlock+0xe/0x10
[ 3031.280059]  [<ffffffff81732a37>] default_device_exit_batch+0x147/0x170
[ 3031.280059]  [<ffffffff810c4140>] ? __wake_up_sync+0x20/0x20
[ 3031.280059]  [<ffffffff8172a892>] ops_exit_list.isra.4+0x52/0x60
[ 3031.280059]  [<ffffffff8172b8e2>] cleanup_net+0x1c2/0x2a0
[ 3031.280059]  [<ffffffff8109a585>] process_one_work+0x165/0x480
[ 3031.280059]  [<ffffffff8109a8eb>] worker_thread+0x4b/0x4c0
[ 3031.280059]  [<ffffffff8109a8a0>] ? process_one_work+0x480/0x480
[ 3031.280059]  [<ffffffff810a0c25>] kthread+0xe5/0x100
[ 3031.280059]  [<ffffffff810a0b40>] ? kthread_create_on_node+0x1e0/0x1e0
[ 3031.280059]  [<ffffffff81840f0f>] ret_from_fork+0x3f/0x70
[ 3031.280059]  [<ffffffff810a0b40>] ? kthread_create_on_node+0x1e0/0x1e0
[ 3031.280059] Code: 44 00 00 55 48 8d 47 08 48 89 e5 41 54 53 48 39 47 08 48 
89 fb 4c 8b 27 75 3e 48 83 7b 18 00 75 4f 48 83 bb b8 02 00 00 00 75 60 <49> 8b 
84 24 48 04 00 00 65 ff 08 8b 83 58 01 00 00 85 c0 74 5e
[ 3031.280059] RIP  [<ffffffff8181b9cd>] in6_dev_finish_destroy+0x2d/0xc0
[ 3031.280059]  RSP <ffff880043383bf8>
[ 3031.280059] CR2: 0000000000000448
[ 3031.280059] ---[ end trace 5d921dfe814c9c7e ]---

** Summary changed:

- 4.4.0-83-generic + Docker + EC2 t2.small frequently crashes at cgroup_rmdir 
GPF
+ 4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1702665

Title:
  4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF

Status in docker package in Ubuntu:
  New
Status in linux package in Ubuntu:
  Confirmed

Bug description:
  We run xenial-based Docker container hosts on EC2 with Amazon ECS.
  Recently we refreshed our base image, we started to see frequent
  panic.

  Hosts run Amazon ECS Agent, and the agent automatically creates or
  destroys Docker container based on requests onto ECS cluster.

  I think this crash is caused by Docker-related operations, because
  crashing at cgroups.

  Also, we're running several different cluster with another EC2
  instance types, using same image. This problem is only reproducing at
  t2.small instances. (We also run c4.large and m4.* clusters)

  Our previous image ran 4.4.0-79-generic, and we see no problem with
  79.

  [30558.783899] general protection fault: 0000 [#1] SMP 
  [30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment 
xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 
nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack 
x_tables nf_nat nf_conntrack br_netfilter bridge stp llc isofs ppdev input_leds 
serio_raw parport_pc parport autofs4 btrfs xor raid6_pq crct10dif_pclmul 
crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul 
glue_helper ablk_helper cryptd psmouse floppy
  [30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic 
#106-Ubuntu
  [30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
  [30558.784056] task: ffff88007c4e8000 ti: ffff88007c4e4000 task.ti: 
ffff88007c4e4000
  [30558.784056] RIP: 0010:[<ffffffff811193ff>]  [<ffffffff811193ff>] 
cgroup_destroy_locked+0x5f/0xf0
  [30558.784056] RSP: 0018:ffff88007c4e7e40  EFLAGS: 00010212
  [30558.784056] RAX: ffff8800114481bd RBX: ffff88002827ba50 RCX: 
ffff88007ab8d150
  [30558.784056] RDX: 00111e7e00ffff88 RSI: ffff88002827ba54 RDI: 
ffffffff8217745c
  [30558.784056] RBP: ffff88007c4e7e60 R08: 0000000000000020 R09: 
ffff88007c4e7e70
  [30558.784056] R10: 000000000637760b R11: ffff880011829a80 R12: 
ffff88007ab8d000
  [30558.784056] R13: 0000000000000000 R14: 0000559b48b2dcc0 R15: 
00000000ffffff9c
  [30558.784056] FS:  00007f29cf0db8c0(0000) GS:ffff88007d200000(0000) 
knlGS:0000000000000000
  [30558.784056] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  [30558.784056] CR2: 00007fa466d58180 CR3: 000000007c190000 CR4: 
00000000001406f0
  [30558.784056] Stack:
  [30558.784056]  ffff88002827ba50 ffff88002827ba50 ffff8800373e70d0 
0000559b48b2dcc0
  [30558.784056]  ffff88007c4e7e80 ffffffff811194b3 ffff88002827ba50 
0000000000000000
  [30558.784056]  ffff88007c4e7ea0 ffffffff8128ddcd ffff880011829a80 
0000000000000000
  [30558.784056] Call Trace:
  [30558.784056]  [<ffffffff811194b3>] cgroup_rmdir+0x23/0x40
  [30558.784056]  [<ffffffff8128ddcd>] kernfs_iop_rmdir+0x4d/0x80
  [30558.784056]  [<ffffffff8121b134>] vfs_rmdir+0xb4/0x130
  [30558.784056]  [<ffffffff8121f83f>] do_rmdir+0x1df/0x200
  [30558.784056]  [<ffffffff81220546>] SyS_rmdir+0x16/0x20
  [30558.784056]  [<ffffffff81840b72>] entry_SYSCALL_64_fastpath+0x16/0x71
  [30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50 
01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 <c6> 82 
b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49 
  [30558.784056] RIP  [<ffffffff811193ff>] cgroup_destroy_locked+0x5f/0xf0
  [30558.784056]  RSP <ffff88007c4e7e40>
  [30558.960828] ---[ end trace 7634e03ff94e8934 ]---
  [30558.964811] Kernel panic - not syncing: Fatal exception in interrupt
  [30558.968805] Kernel Offset: disabled

  ProblemType: Bug
  DistroRelease: Ubuntu 16.04
  Package: linux-image-4.4.0-83-generic 4.4.0-83.106
  ProcVersionSignature: Ubuntu 4.4.0-83.106-generic 4.4.70
  Uname: Linux 4.4.0-83-generic x86_64
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Jul  6 10:22 seq
   crw-rw---- 1 root audio 116, 33 Jul  6 10:22 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.1-0ubuntu2.6
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
'/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  Date: Thu Jul  6 10:27:37 2017
  Ec2AMI: ami-34100353
  Ec2AMIManifest: (unknown)
  Ec2AvailabilityZone: ap-northeast-1c
  Ec2InstanceType: t2.small
  Ec2Kernel: unavailable
  Ec2Ramdisk: unavailable
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lsusb: Error: command ['lsusb'] failed with exit code 1:
  MachineType: Xen HVM domU
  PciMultimedia:
   
  ProcEnviron:
   SHELL=/bin/bash
   TERM=screen-256color
   PATH=(custom, no user)
   LANG=en_US.UTF-8
  ProcFB:
   
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-83-generic 
root=UUID=f76be987-234f-4071-87d4-06318cfc2135 ro cgroup_enable=memory 
swapaccount=1 console=tty1 console=ttyS0
  RelatedPackageVersions:
   linux-restricted-modules-4.4.0-83-generic N/A
   linux-backports-modules-4.4.0-83-generic  N/A
   linux-firmware                            N/A
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  SourcePackage: linux
  UpgradeStatus: No upgrade log present (probably fresh install)
  WifiSyslog:
   
  dmi.bios.date: 02/16/2017
  dmi.bios.vendor: Xen
  dmi.bios.version: 4.2.amazon
  dmi.chassis.type: 1
  dmi.chassis.vendor: Xen
  dmi.modalias: 
dmi:bvnXen:bvr4.2.amazon:bd02/16/2017:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
  dmi.product.name: HVM domU
  dmi.product.version: 4.2.amazon
  dmi.sys.vendor: Xen

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/docker/+bug/1702665/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to