After a lot of debugging it turns out that the bug is in igb... driver: igb version: 5.6.0-k firmware-version: 0. 6-1
03:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) It's interesting that it only seems to happen on longer links... Any clues? On Sat, Jul 11, 2020 at 5:53 PM Ian Kumlien <ian.kuml...@gmail.com> wrote: > > Hi, > > I first detected this with 5.7.6 but it seems to apply as far back as 5.6.1... > (so, 5.7.8 client -> nat (5.6.1 -> 5.8-rc4 -> server 5.7.7) > > It seems to me that the window size doesn't advance, so i did revert > the tcp: grow window for OOO packets only for SACK flows [1] > but it did no difference... > > I have a 384 MB tcpdump of a iperf3 session that starts low and then > actually starts to get the bandwidth... > I do use BBR - I have tried with cubic... it didn't help - the NAT > machine does use fq but changing it doesn't seem to yield any other > results. > > Doing -P10 gives you the bandwith and can sometimes break the > stalemate but you'll end up back with the lower transfer speed again. > (it only seems to apply to NAT - the machine is a: A2SDi-12C-HLN4F and > has handled this without problems in the past...) > > > [1] > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=v5.7.8&id=bf780119617797b5690e999e59a64ad79a572374 > > First iperf3 as a reference: > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 113 MBytes 945 Mbits/sec 0 814 KBytes > [ 5] 1.00-2.00 sec 109 MBytes 912 Mbits/sec 0 806 KBytes > [ 5] 2.00-3.00 sec 112 MBytes 944 Mbits/sec 31 792 KBytes > [ 5] 3.00-4.00 sec 101 MBytes 849 Mbits/sec 31 1.18 MBytes > [ 5] 4.00-5.00 sec 108 MBytes 902 Mbits/sec 0 783 KBytes > [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 31 778 KBytes > [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 93 772 KBytes > [ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 778 KBytes > [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 60 778 KBytes > [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 92 814 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 1.07 GBytes 923 Mbits/sec 338 sender > [ 5] 0.00-10.01 sec 1.07 GBytes 919 Mbits/sec receiver > > After that: > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 4.77 MBytes 40.0 Mbits/sec 0 42.4 KBytes > [ 5] 1.00-2.00 sec 4.10 MBytes 34.4 Mbits/sec 0 84.8 KBytes > [ 5] 2.00-3.00 sec 4.60 MBytes 38.6 Mbits/sec 0 87.7 KBytes > [ 5] 3.00-4.00 sec 4.23 MBytes 35.4 Mbits/sec 0 42.4 KBytes > [ 5] 4.00-5.00 sec 4.23 MBytes 35.4 Mbits/sec 0 42.4 KBytes > [ 5] 5.00-6.00 sec 4.47 MBytes 37.5 Mbits/sec 0 76.4 KBytes > [ 5] 6.00-7.00 sec 5.47 MBytes 45.9 Mbits/sec 0 67.9 KBytes > [ 5] 7.00-8.00 sec 4.66 MBytes 39.1 Mbits/sec 0 67.9 KBytes > [ 5] 8.00-9.00 sec 4.35 MBytes 36.5 Mbits/sec 0 82.0 KBytes > [ 5] 9.00-10.00 sec 4.66 MBytes 39.1 Mbits/sec 0 139 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 45.5 MBytes 38.2 Mbits/sec 0 sender > [ 5] 0.00-10.00 sec 45.0 MBytes 37.8 Mbits/sec receiver > > You even get some: > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.00 sec 5.38 MBytes 45.2 Mbits/sec 0 42.4 KBytes > [ 5] 1.00-2.00 sec 7.08 MBytes 59.4 Mbits/sec 0 535 KBytes > [ 5] 2.00-3.00 sec 108 MBytes 907 Mbits/sec 0 778 KBytes > [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 814 KBytes > [ 5] 4.00-5.00 sec 91.2 MBytes 765 Mbits/sec 0 829 KBytes > [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 783 KBytes > [ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 769 KBytes > [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 778 KBytes > [ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 809 KBytes > [ 5] 9.00-10.00 sec 110 MBytes 923 Mbits/sec 0 823 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 879 MBytes 738 Mbits/sec 0 sender > [ 5] 0.00-10.00 sec 875 MBytes 734 Mbits/sec receiver
