On 2019/04/05 16:14, Felix Fietkau wrote: > On 2019-04-05 09:11, Rafał Miłecki wrote: >> On 05.04.2019 07:48, Rafał Miłecki wrote: >>> On 05.04.2019 06:26, Toshiaki Makita wrote: >>>> My test results: >>>> >>>> Receiving packets from eth0.10, forwarding them to eth0.20 and applying >>>> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. >>>> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). >>>> Measured TCP throughput by netperf. >>>> >>>> GRO on : 17 Gbps >>>> GRO off: 5 Gbps >>>> >>>> So I failed to reproduce your problem. >>> >>> :( Thanks for trying & checking that! >>> >>> >>>> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar >>>> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on >>>> your machine? >>> >>> 1) ethtool -K eth0 gro on + iperf running (577 Mb/s) >>> root@OpenWrt:/# mpstat -P ALL 10 3 >>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>> >>> 16:33:40 CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> 16:33:50 all 0.00 0.00 0.00 0.00 0.00 58.79 0.00 >>> 0.00 41.21 >>> 16:33:50 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 >>> 0.00 0.00 >>> 16:33:50 1 0.00 0.00 0.00 0.00 0.00 17.58 0.00 >>> 0.00 82.42 >>> >>> 16:33:50 CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> 16:34:00 all 0.00 0.00 0.05 0.00 0.00 59.44 0.00 >>> 0.00 40.51 >>> 16:34:00 0 0.00 0.00 0.10 0.00 0.00 99.90 0.00 >>> 0.00 0.00 >>> 16:34:00 1 0.00 0.00 0.00 0.00 0.00 18.98 0.00 >>> 0.00 81.02 >>> >>> 16:34:00 CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> 16:34:10 all 0.00 0.00 0.00 0.00 0.00 59.59 0.00 >>> 0.00 40.41 >>> 16:34:10 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 >>> 0.00 0.00 >>> 16:34:10 1 0.00 0.00 0.00 0.00 0.00 19.18 0.00 >>> 0.00 80.82 >>> >>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> Average: all 0.00 0.00 0.02 0.00 0.00 59.27 0.00 >>> 0.00 40.71 >>> Average: 0 0.00 0.00 0.03 0.00 0.00 99.97 0.00 >>> 0.00 0.00 >>> Average: 1 0.00 0.00 0.00 0.00 0.00 18.58 0.00 >>> 0.00 81.42 >>> >>> >>> 2) ethtool -K eth0 gro off + iperf running (941 Mb/s) >>> root@OpenWrt:/# mpstat -P ALL 10 3 >>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>> >>> 16:34:39 CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> 16:34:49 all 0.00 0.00 0.05 0.00 0.00 86.91 0.00 >>> 0.00 13.04 >>> 16:34:49 0 0.00 0.00 0.10 0.00 0.00 78.22 0.00 >>> 0.00 21.68 >>> 16:34:49 1 0.00 0.00 0.00 0.00 0.00 95.60 0.00 >>> 0.00 4.40 >>> >>> 16:34:49 CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> 16:34:59 all 0.00 0.00 0.10 0.00 0.00 87.06 0.00 >>> 0.00 12.84 >>> 16:34:59 0 0.00 0.00 0.20 0.00 0.00 79.72 0.00 >>> 0.00 20.08 >>> 16:34:59 1 0.00 0.00 0.00 0.00 0.00 94.41 0.00 >>> 0.00 5.59 >>> >>> 16:34:59 CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> 16:35:09 all 0.00 0.00 0.05 0.00 0.00 85.71 0.00 >>> 0.00 14.24 >>> 16:35:09 0 0.00 0.00 0.10 0.00 0.00 79.42 0.00 >>> 0.00 20.48 >>> 16:35:09 1 0.00 0.00 0.00 0.00 0.00 92.01 0.00 >>> 0.00 7.99 >>> >>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> Average: all 0.00 0.00 0.07 0.00 0.00 86.56 0.00 >>> 0.00 13.37 >>> Average: 0 0.00 0.00 0.13 0.00 0.00 79.12 0.00 >>> 0.00 20.75 >>> Average: 1 0.00 0.00 0.00 0.00 0.00 94.01 0.00 >>> 0.00 5.99 >>> >>> >>> 3) System idle (no iperf) >>> root@OpenWrt:/# mpstat -P ALL 10 1 >>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>> >>> 16:35:31 CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> 16:35:41 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 0.00 100.00 >>> 16:35:41 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 0.00 100.00 >>> 16:35:41 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 0.00 100.00 >>> >>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal >>> %guest %idle >>> Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 0.00 100.00 >>> Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 0.00 100.00 >>> Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >>> 0.00 100.00 >>> >>> >>>> If CPU is 100%, perf may help us analyze your problem. If it's >>>> available, try running below while testing: >>>> # perf record -a -g -- sleep 5 >>>> >>>> And then run this after testing: >>>> # perf report --no-child >>> >>> I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now. >> >> I guess its GRO + csum_partial() to be blamed for this performance drop. >> >> Maybe csum_partial() is very fast on your powerful machine and few extra >> calls >> don't make a difference? I can imagine it affecting much slower home router >> with >> ARM cores. > Most high performance Ethernet devices implement hardware checksum > offload, which completely gets rid of this overhead. > Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is > why you're getting such crappy performance.
Hmm... now I disabled rx checksum and tried the test again, and indeed I see csum_partial from GRO path. But I also see csum_partial even without GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete. Probably Rafał disabled nf_conntrack_checksum sysctl knob? But anyway even with disabling rx csum offload my machine has better performance with GRO. I'm sure in some cases GRO should be disabled, but I guess it's difficult to determine whether we should disable GRO or not automatically when csum offload is not available. -- Toshiaki Makita
