** Description changed: Ubuntu linux distro, 4.15.0-62 kernel, server platform. This OS is used as an IPSec VPN gateway. It serves up to several hundred concurrent connections In an attempt to upgrade from the 4.4 kernel to 4.15, the team noticed that VPN gateway VMs were running out of physical memory after 12-48 hours, depending on load. Attachments from a server machine in this state in attached leakinfo.txt output of free -t output of /proc/meminfo in out of memory condition - output of /slabtop -o -sc + output of /slabtop -o -sc /sys/kernel/debug/page_owner sorted and aggregated after server ran for 12 hrs and ran out of memory Patches for 4.15 and 5.4 Highlight from page_owner, we can see the leak is a buffer associated with the ipsec impelementation. Each connection leaks 32k of memory via alloc_page with order=3 + 100960 times: Page allocated via order 3, mask 0x1085220(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_COMP) - get_page_from_freelist+0xd64/0x1250 - __alloc_pages_nodemask+0x11c/0x2e0 - alloc_pages_current+0x6a/0xe0 - skb_page_frag_refill+0x71/0x100 - esp_output_head+0x265/0x3e0 [esp4] - esp_output+0xbc/0x180 [esp4] - xfrm_output_resume+0x179/0x530 - xfrm_output+0x8e/0x230 - xfrm4_output_finish+0x2b/0x30 - __xfrm4_output+0x3a/0x50 - xfrm4_output+0x43/0xc0 - ip_forward_finish+0x51/0x80 - ip_forward+0x38a/0x480 - ip_rcv_finish+0x122/0x410 - ip_rcv+0x292/0x360 - __netif_receive_skb_core+0x815/0xbd0 + get_page_from_freelist+0xd64/0x1250 + __alloc_pages_nodemask+0x11c/0x2e0 + alloc_pages_current+0x6a/0xe0 + skb_page_frag_refill+0x71/0x100 + esp_output_head+0x265/0x3e0 [esp4] + esp_output+0xbc/0x180 [esp4] + xfrm_output_resume+0x179/0x530 + xfrm_output+0x8e/0x230 + xfrm4_output_finish+0x2b/0x30 + __xfrm4_output+0x3a/0x50 + xfrm4_output+0x43/0xc0 + ip_forward_finish+0x51/0x80 + ip_forward+0x38a/0x480 + ip_rcv_finish+0x122/0x410 + ip_rcv+0x292/0x360 + __netif_receive_skb_core+0x815/0xbd0 Patch to fix this issue in 4.15 (tested and verified on same server exhibiting above leak): diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index 728272f..7842f83 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -451,6 +451,10 @@ static void xfrm_state_gc_destroy(struct xfrm_state *x) - } - xfrm_dev_state_free(x); - security_xfrm_state_free(x); + } + xfrm_dev_state_free(x); + security_xfrm_state_free(x); + + if(x->xfrag.page) + put_page(x->xfrag.page); + - kfree(x); + kfree(x); } - - - Patch for master branch (5.4 I believe) from Paul Wouters (p...@nohats.ca) + Patch for master branch (5.4 I believe) from Paul Wouters + (p...@nohats.ca) diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c index c6f3c4a1bd99..f3423562d933 100644 --- a/net/xfrm/xfrm_state.c +++ b/net/xfrm/xfrm_state.c @@ -495,6 +495,8 @@ static void ___xfrm_state_destroy(struct xfrm_state *x) - x->type->destructor(x); - xfrm_put_type(x->type); - } + x->type->destructor(x); + xfrm_put_type(x->type); + } + if (x->xfrag.page) + put_page(x->xfrag.page); - xfrm_dev_state_free(x); - security_xfrm_state_free(x); - xfrm_state_free(x); - + xfrm_dev_state_free(x); + security_xfrm_state_free(x); + xfrm_state_free(x); Severity: Critical - we are unable to use any kernel later than 4.11, and are sticking with 4.4 in production.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1853197 Title: Memory leak in net/xfrm/xfrm_state.c - 8 pages per ipsec connection To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1853197/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
