On 2017年11月28日 00:21, Wei Xu wrote:
On Mon, Nov 20, 2017 at 02:25:17PM -0500, Matthew Rosato wrote:
On 11/14/2017 03:11 PM, Matthew Rosato wrote:
On 11/12/2017 01:34 PM, Wei Xu wrote:
On Sat, Nov 11, 2017 at 03:59:54PM -0500, Matthew Rosato wrote:
This case should be quite similar with pkgten, if you got improvement with
pktgen, usually it was also the same for UDP, could you please try to disable
tso, gso, gro, ufo on all host tap devices and guest virtio-net devices?
Currently
the most significant tests would be like this AFAICT:
Host->VM 4.12 4.13
TCP:
UDP:
pktgen:
So, I automated these scenarios for extended overnight runs and started
experiencing OOM conditions overnight on a 40G system. I did a bisect
and it also points to c67df11f. I can see a leak in at least all of the
Host->VM testcases (TCP, UDP, pktgen), but the pktgen scenario shows the
fastest leak.
I enabled slub_debug on base 4.13 and ran my pktgen scenario in short
intervals until a large% of host memory was consumed. Numbers below
after the last pktgen run completed. The summary is that a very large #
of active skbuff_head_cache entries can be seen - The sum of alloc/free
calls match up, but the # of active skbuff_head_cache entries keeps
growing each time the workload is run and never goes back down in
between runs.
free -h:
total used free shared buff/cache available
Mem: 39G 31G 6.6G 472K 1.4G 6.8G
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
1001952 1000610 99% 0.75K 23856 42 763392K skbuff_head_cache
126192 126153 99% 0.36K 2868 44 45888K ksm_rmap_item
100485 100435 99% 0.41K 1305 77 41760K kernfs_node_cache
63294 39598 62% 0.48K 959 66 30688K dentry
31968 31719 99% 0.88K 888 36 28416K inode_cache
/sys/kernel/slab/skbuff_head_cache/alloc_calls :
259 __alloc_skb+0x68/0x188 age=1/135076/135741 pid=0-11776 cpus=0,2,4,18
1000351 __build_skb+0x42/0xb0 age=8114/63172/117830 pid=0-11863 cpus=0,10
/sys/kernel/slab/skbuff_head_cache/free_calls:
13492 <not-available> age=4295073614 pid=0 cpus=0
978298 tun_do_read.part.10+0x18c/0x6a0 age=8532/63624/110571 pid=11733
cpus=1-19
6 skb_free_datagram+0x32/0x78 age=11648/73253/110173 pid=11325
cpus=4,8,10,12,14
3 __dev_kfree_skb_any+0x5e/0x70 age=108957/115043/118269
pid=0-11605 cpus=5,7,12
1 netlink_broadcast_filtered+0x172/0x470 age=136165 pid=1 cpus=4
2 netlink_dump+0x268/0x2a8 age=73236/86857/100479 pid=11325 cpus=4,12
1 netlink_unicast+0x1ae/0x220 age=12991 pid=9922 cpus=12
1 tcp_recvmsg+0x2e2/0xa60 age=0 pid=11776 cpus=6
3 unix_stream_read_generic+0x810/0x908 age=15443/50904/118273
pid=9915-11581 cpus=8,16,18
2 tap_do_read+0x16a/0x488 [tap] age=42338/74246/106155
pid=11605-11699 cpus=2,9
1 macvlan_process_broadcast+0x17e/0x1e0 [macvlan] age=18835
pid=331 cpus=11
8800 pktgen_thread_worker+0x80a/0x16d8 [pktgen] age=8545/62184/110571
pid=11863 cpus=0
By comparison, when running 4.13 with c67df11f reverted, here's the same
output after the exact same test:
free -h:
total used free shared buff/cache available
Mem: 39G 783M 37G 472K 637M 37G
slabtop:
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
714 256 35% 0.75K 17 42 544K skbuff_head_cache
/sys/kernel/slab/skbuff_head_cache/alloc_calls:
257 __alloc_skb+0x68/0x188 age=0/65252/65507 pid=1-11768 cpus=10,15
/sys/kernel/slab/skbuff_head_cache/free_calls:
255 <not-available> age=4295003081 pid=0 cpus=0
1 netlink_broadcast_filtered+0x2e8/0x4e0 age=65601 pid=1 cpus=15
1 tcp_recvmsg+0x2e2/0xa60 age=0 pid=11768 cpus=16
Thanks a lot for the test, and sorry for the late update, I was working on
the code path and didn't find anything helpful to you till today.
I did some tests and initially it turned out that the bottleneck was the guest
kernel stack(napi) side, followed by tracking the traffic footprints and it
appeared as the loss happened when vring was full and could not be drained
out by the guest, afterwards it triggered a SKB drop in vhost driver due
to no headcount to fill it with, it can be avoided by deferring consuming the
SKB after having obtained a sufficient headcount with below patch.
Could you please try it? It is based on 4.13 and I also applied Jason's
'conditionally enable tx polling' patch.
https://lkml.org/lkml/2016/6/1/39
This patch has already been merged.
I only tested one instance case from Host -> VM with uperf & iperf3, I like
iperf3 a bit more since it spontaneously tells the retransmitted and cwnd
during testing. :)
To maximize the performance of one instance case, two vcpus are needed,
one does the kernel napi and the other one should serve the socket syscall
(mostly reading) from uperf/iperf userspace, so I set two vcpus to the guest
and pinned the iperf/uperf slave to the one not used by kernel napi, you may
need to check out which one you should pin properly by seeing the CPU
utilization with a quick trial test before running the long duration test.
Slight performance improvement for tcp with the patch(host/guest offload off)
on x86, also 4.12 wins the game with 20-30% possibility from time to time, but
the cwnd and retransmitted statistics are almost the same now, the 'retrans'
was about 10x times more and cwnd was 6x smaller than 4.12 before.
Here is one typical sample of my tests.
4.12 4.13
offload on: 36.8Gbits 37.4Gbits
offload off: 7.68Gbits 7.84Gbits
I also borrowed a s390x machine with 6 cpus and 4G memory from system z team,
it seems 4.12 is still a bit faster than 4.13, could you please see if this
is aligned with your test bed?
4.12 4.13
offload on: 37.3Gbits 38.3Gbits
offload off: 6.26Gbits 6.06Gbits
For pktgen, I got 10% improvement(xdp1 drop on guest) which is a bit faster
than Jason's number before.
4.12 4.13
3.33 Mpss 3.70 Mpps
Thanks again for all the tests your have done.
Wei
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -776,8 +776,6 @@ static void handle_rx(struct vhost_net *net)
/* On error, stop handling until the next kick. */
if (unlikely(headcount < 0))
goto out;
- if (nvq->rx_array)
- msg.msg_control = vhost_net_buf_consume(&nvq->rxq);
/* On overrun, truncate and discard */
if (unlikely(headcount > UIO_MAXIOV)) {
I think you need do msg.msg_control = vhost_net_buf_consume() here too.
iov_iter_init(&msg.msg_iter, READ, vq->iov, 1, 1);
@@ -798,6 +796,10 @@ static void handle_rx(struct vhost_net *net)
* they refilled. */
goto out;
}
+
+ if (nvq->rx_array)
+ msg.msg_control = vhost_net_buf_consume(&nvq->rxq);
+
/* We don't need to be notified again. */
iov_iter_init(&msg.msg_iter, READ, vq->iov, in, vhost_len);
fixup = msg.msg_iter;
Good catch, this fixes the memory leak too.
I suggest to post a formal patch for -net as soon as possible too since
it was a valid fix even if it does not help for performance.
Thanks
