Andrey, The patches seem to be against infiniband drivers. Would I get any value from trying the elrepo 3.17.3 kernel to hopefully pick up the compaction changes?
Regards Darryl ________________________________________ From: Andrey Korolyov <and...@xdel.ru> Sent: Friday, 21 November 2014 8:27 AM To: Bond, Darryl Subject: Re: [ceph-users] Kernel memory allocation oops Centos 7 Replying off-list: please check ongoing thread there: http://marc.info/?l=linux-mm&m=141643863320754&w=2 and, if possible, check proposed patches against your env. Your case a bit different because you are using bnx2x and not ipoib which was suggested as a potential troublemaker. Sorry, just clicked on the google invitation while scratching my eye, please ignore it (blaming Gmail interface). On Fri, Nov 21, 2014 at 1:10 AM, Bond, Darryl <db...@nrggos.com.au> wrote: > Brief outline: > > 6 Node production cluster. Each node Dell R610, 8x1.4TB SAS Disks, Samsung > M.2 PCIe SSD for journals, 32GB RAM, Broadcom 10G interfaces. > > Ceph 0.80.7-0.el7.centos from the ceph repositories. > > > About 10 times per day, each node will oops with the following message: > > An example: > > Nov 21 07:07:50 ceph14-04 kernel: warn_alloc_failed: 366 callbacks suppressed > Nov 21 07:07:50 ceph14-04 kernel: swapper/4: page allocation failure: > order:2, mode:0x104020 > Nov 21 07:07:50 ceph14-04 kernel: kswapd0: page allocation failure: order:2, > mode:0x104020 > Nov 21 07:07:50 ceph14-04 kernel: CPU: 5 PID: 176 Comm: kswapd0 Not tainted > 3.10.0-123.9.3.el7.x86_64 #1 > Nov 21 07:07:50 ceph14-04 kernel: Hardware name: Dell Inc. PowerEdge > R620/01W23F, BIOS 2.2.2 01/16/2014 > Nov 21 07:07:50 ceph14-04 kernel: ceph-osd: page allocation failure: order:2, > mode:0x104020 > Nov 21 07:07:50 ceph14-04 kernel: systemd-journal: page allocation failure: > order:2, mode:0x104020 > Nov 21 07:07:50 ceph14-04 kernel: CPU: 9 PID: 704 Comm: systemd-journal Not > tainted 3.10.0-123.9.3.el7.x86_64 #1 > Nov 21 07:07:50 ceph14-04 kernel: CPU: 4 PID: 0 Comm: swapper/4 Not tainted > 3.10.0-123.9.3.el7.x86_64 #1 > Nov 21 07:07:50 ceph14-04 kernel: Hardware name: Dell Inc. PowerEdge > R620/01W23F, BIOS 2.2.2 01/16/2014 > Nov 21 07:07:50 ceph14-04 kernel: 0000000000104020 000000005835f665 > ffff88080f0a3a00 ffffffff815e239b > Nov 21 07:07:50 ceph14-04 kernel: ceph-osd: page allocation failure: order:2, > mode:0x104020 > Nov 21 07:07:50 ceph14-04 kernel: Hardware name: Dell Inc. PowerEdge > R620/01W23F, BIOS 2.2.2 01/16/2014 > Nov 21 07:07:50 ceph14-04 kernel: 0000000000104020 > Nov 21 07:07:50 ceph14-04 kernel: CPU: 0 PID: 7453 Comm: ceph-osd Not tainted > 3.10.0-123.9.3.el7.x86_64 #1 > Nov 21 07:07:50 ceph14-04 kernel: ffff88080f0a3a90 > Nov 21 07:07:50 ceph14-04 kernel: 0000000000104020 > Nov 21 07:07:50 ceph14-04 kernel: 000000009c9142fd > Nov 21 07:07:50 ceph14-04 kernel: Hardware name: Dell Inc. PowerEdge > R620/01W23F, BIOS 2.2.2 01/16/2014 > > or another example: > > Nov 20 09:03:09 ceph14-06 kernel: warn_alloc_failed: 3803 callbacks suppressed > Nov 20 09:03:09 ceph14-06 kernel: swapper/11: page allocation failure: > order:2, mode:0x104020 > Nov 20 09:03:09 ceph14-06 kernel: CPU: 11 PID: 0 Comm: swapper/11 Not tainted > 3.10.0-123.9.3.el7.x86_64 #1 > Nov 20 09:03:09 ceph14-06 kernel: Hardware name: Dell Inc. PowerEdge > R620/01W23F, BIOS 2.2.2 01/16/2014 > Nov 20 09:03:09 ceph14-06 kernel: 0000000000104020 dbf4eb51672ffc35 > ffff88080f163a00 ffffffff815e239b > Nov 20 09:03:09 ceph14-06 kernel: ffff88080f163a90 ffffffff81147340 > 0000000000000002 ffff88080f163a50 > Nov 20 09:03:09 ceph14-06 kernel: ffff88082ffd7e80 ffff88082ffd7e80 > 0000000000000002 dbf4eb51672ffc35 > Nov 20 09:03:09 ceph14-06 kernel: Call Trace: > Nov 20 09:03:09 ceph14-06 kernel: <IRQ> [<ffffffff815e239b>] > dump_stack+0x19/0x1b > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff81147340>] > warn_alloc_failed+0x110/0x180 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff8114b4dc>] > __alloc_pages_nodemask+0x90c/0xb10 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff8150941d>] ? > ip_rcv_finish+0x7d/0x350 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff81509ce4>] ? ip_rcv+0x234/0x380 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff814d01c0>] ? > netif_receive_skb+0x40/0xd0 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff81188349>] > alloc_pages_current+0xa9/0x170 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff8114629e>] > __get_free_pages+0xe/0x50 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff811930ee>] > kmalloc_order_trace+0x2e/0xa0 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff814cfb32>] ? > __netif_receive_skb_core+0x282/0x870 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffff81194749>] __kmalloc+0x219/0x230 > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffffa0145bca>] > bnx2x_frag_alloc.isra.65+0x2a/0x40 [bnx2x] > Nov 20 09:03:09 ceph14-06 kernel: [<ffffffffa01461d4>] > bnx2x_alloc_rx_data.isra.72+0x54/0x1c0 [bnx2x] > Nov 20 09:03:09 ceph14-06 kernel: swapper/8: page allocation failure: > order:2, mode:0x104020 > > All oops seem to be triggered by page allocation failure. > > The effect of the oops is that the server has memory allocation errors all > over the place , but mainly in the network stack. Not surprising since that > would be the major activity. > I have set vm swappiness to 0 on one node but it still generates the errors. > > Mem: 32732696 32507888 224808 51004 0 26187580 > -/+ buffers/cache: 6320308 26412388 > Swap: 31249404 308396 30941008 > > > > Each oops is serious and affects the machine enough to trip nagios which > scans each 5 minutes. It would appear that the node doesn't respond to the > network for many seconds. > > ? > > > A couple of observations: > Affects mon/osd servers as well as just osd servers, although they don't seem > to be any more or less affected. > > The OSD processes are affected on occasions but they do not seem to be using > excessive memory > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 13571 root 20 0 1847536 581636 4968 S 2.0 1.8 211:09.58 ceph-osd > 13707 root 20 0 1803560 523904 4956 S 2.0 1.6 184:22.69 ceph-osd > 13997 root 20 0 1905820 580768 5088 S 1.7 1.8 182:28.36 ceph-osd > 13436 root 20 0 1783656 544400 5076 S 1.3 1.7 216:53.34 ceph-osd > 13840 root 20 0 1778296 570400 4380 S 1.3 1.7 184:09.06 ceph-osd > 14154 root 20 0 1881804 617748 5460 S 1.3 1.9 227:42.08 ceph-osd > 14356 root 20 0 1906236 593936 4512 S 1.3 1.8 188:28.77 ceph-osd > 14491 root 20 0 1837232 546140 4264 S 1.0 1.7 182:27.13 ceph-osd > > The main culprit seems to be the vm page cache. > > Any recommendations? > > Regards > Darryl > > > > ________________________________ > > The contents of this electronic message and any attachments are intended only > for the addressee and may contain legally privileged, personal, sensitive or > confidential information. If you are not the intended addressee, and have > received this email, any transmission, distribution, downloading, printing or > photocopying of the contents of this message or attachments is strictly > prohibited. Any legal privilege or confidentiality attached to this message > and attachments is not waived, lost or destroyed by reason of delivery to any > person other than intended addressee. If you have received this message and > are not the intended addressee you should notify the sender by return email > and destroy all copies of the message and any attachments. Unless expressly > attributed, the views expressed in this email do not necessarily represent > the views of the company. > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
