> On 16. Mar 2021, at 15:18, Marek Zarychta <zarych...@plan-b.pwste.edu.pl>
> wrote:
>
> W dniu 16.03.2021 o 12:50, tue...@freebsd.org pisze:
>>> On 16. Mar 2021, at 11:55, Blake Hartshorn <cont...@blakehartshorn.com>
>>> wrote:
>>>
>>> Hi Michael,
>>>
>>> I've attached tcpdumps for port 80 on both sides of a bad transfer, using 2
>>> VMs in the same datacenter, FreeBSD 13 serving and 12 as a client. A friend
>>> of mine suggested I also run some tests with iperf3, so pasting those
>>> results below. You'll see it going fast in one direction and crawling in
>>> the other on TCP. There's also some disparity on UDP.
>> The problem is that the server provides TCP segments larger than the MTU
>> to the NIC. These are dropped and needs to be retransmitted. That is why
>> it takes so long. So I guess TSO is enabled on the NIC and not working
>> correctly.
>>
>> What is the output of ifconfig? Can you disable TSO? Does that work around
>> the problem?
>>
>> Best regards
>> Michael
>>>
>>> TCP:
>>> [ ID][Role] Interval Transfer Bitrate Retr Cwnd
>>> [ 5][TX-C] 0.00-1.00 sec 27.7 MBytes 233 Mbits/sec 342 26.7
>>> KBytes
>>> [ 7][RX-C] 0.00-1.00 sec 4.18 KBytes 34.3 Kbits/sec
>>> [ 5][TX-C] 1.00-2.00 sec 15.8 MBytes 132 Mbits/sec 249 52.0
>>> KBytes
>>> [ 7][RX-C] 1.00-2.00 sec 4.18 KBytes 34.3 Kbits/sec
>>> [ 5][TX-C] 2.00-3.00 sec 13.7 MBytes 115 Mbits/sec 307 15.4
>>> KBytes
>>> [ 7][RX-C] 2.00-3.00 sec 4.18 KBytes 34.3 Kbits/sec
>>> [ 5][TX-C] 3.00-4.00 sec 14.5 MBytes 121 Mbits/sec 260 22.4
>>> KBytes
>>> [ 7][RX-C] 3.00-4.00 sec 4.18 KBytes 34.3 Kbits/sec
>>> [ 5][TX-C] 4.00-5.00 sec 14.3 MBytes 120 Mbits/sec 240 37.9
>>> KBytes
>>> [ 7][RX-C] 4.00-5.00 sec 5.58 KBytes 45.7 Kbits/sec
>>> [ 5][TX-C] 5.00-6.00 sec 17.7 MBytes 149 Mbits/sec 363 15.4
>>> KBytes
>>> [ 7][RX-C] 5.00-6.00 sec 4.18 KBytes 34.3 Kbits/sec
>>> [ 5][TX-C] 6.00-7.00 sec 14.8 MBytes 124 Mbits/sec 287 8.38
>>> KBytes
>>> [ 7][RX-C] 6.00-7.00 sec 5.58 KBytes 45.7 Kbits/sec
>>> [ 5][TX-C] 7.00-8.00 sec 14.7 MBytes 123 Mbits/sec 293 28.1
>>> KBytes
>>> [ 7][RX-C] 7.00-8.00 sec 4.18 KBytes 34.3 Kbits/sec
>>> [ 5][TX-C] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 325 18.3
>>> KBytes
>>> [ 7][RX-C] 8.00-9.00 sec 4.18 KBytes 34.3 Kbits/sec
>>> [ 5][TX-C] 9.00-10.00 sec 14.3 MBytes 120 Mbits/sec 315 39.3
>>> KBytes
>>> [ 7][RX-C] 9.00-10.00 sec 4.18 KBytes 34.3 Kbits/sec
>>> - - - - - - - - - - - - - - - - - - - - - - - - -
>>> [ ID][Role] Interval Transfer Bitrate Retr
>>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec 2981
>>> sender
>>> [ 5][TX-C] 0.00-10.00 sec 159 MBytes 134 Mbits/sec
>>> receiver
>>> [ 7][RX-C] 0.00-10.00 sec 77.0 KBytes 63.1 Kbits/sec 65
>>> sender
>>> [ 7][RX-C] 0.00-10.00 sec 44.6 KBytes 36.6 Kbits/sec
>>> receiver
>>> -----------------------------------------------------------------------------------------------------------
>>>
>>>
>>> UDP:
>>> [ ID][Role] Interval Transfer Bitrate Jitter
>>> Lost/Total Datagrams
>>> [ 5][TX-C] 0.00-1.00 sec 81.6 MBytes 685 Mbits/sec 67798
>>>
>>> [ 7][RX-C] 0.00-1.00 sec 8.80 MBytes 73.8 Mbits/sec 0.255 ms
>>> 54070/60475 (89%)
>>> [ 5][TX-C] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec 64802
>>>
>>> [ 7][RX-C] 1.00-2.00 sec 8.52 MBytes 71.5 Mbits/sec 0.154 ms
>>> 68912/75116 (92%)
>>> [ 5][TX-C] 2.00-3.00 sec 73.7 MBytes 618 Mbits/sec 64158
>>>
>>> [ 7][RX-C] 2.00-3.00 sec 8.52 MBytes 71.5 Mbits/sec 0.276 ms
>>> 67738/73945 (92%)
>>> [ 5][TX-C] 3.00-4.00 sec 76.6 MBytes 643 Mbits/sec 63521
>>>
>>> [ 7][RX-C] 3.00-4.00 sec 8.55 MBytes 71.8 Mbits/sec 0.160 ms
>>> 68647/74874 (92%)
>>> [ 5][TX-C] 4.00-5.00 sec 76.1 MBytes 638 Mbits/sec 64614
>>>
>>> [ 7][RX-C] 4.00-5.00 sec 8.55 MBytes 71.7 Mbits/sec 0.461 ms
>>> 67542/73767 (92%)
>>> [ 5][TX-C] 5.00-6.00 sec 75.9 MBytes 637 Mbits/sec 64834
>>>
>>> [ 7][RX-C] 5.00-6.00 sec 8.57 MBytes 71.9 Mbits/sec 0.297 ms
>>> 71565/77806 (92%)
>>> [ 5][TX-C] 6.00-7.00 sec 73.0 MBytes 613 Mbits/sec 63639
>>>
>>> [ 7][RX-C] 6.00-7.00 sec 8.40 MBytes 70.5 Mbits/sec 0.199 ms
>>> 69545/75663 (92%)
>>> [ 5][TX-C] 7.00-8.00 sec 74.6 MBytes 626 Mbits/sec 65030
>>>
>>> [ 7][RX-C] 7.00-8.00 sec 8.78 MBytes 73.6 Mbits/sec 0.254 ms
>>> 67173/73566 (91%)
>>> [ 5][TX-C] 8.00-9.00 sec 75.0 MBytes 629 Mbits/sec 64848
>>>
>>> [ 7][RX-C] 8.00-9.00 sec 8.77 MBytes 73.5 Mbits/sec 0.298 ms
>>> 70932/77315 (92%)
>>> [ 5][TX-C] 9.00-10.00 sec 74.5 MBytes 625 Mbits/sec 64487
>>>
>>> [ 7][RX-C] 9.00-10.00 sec 8.71 MBytes 73.1 Mbits/sec 0.185 ms
>>> 68268/74612 (91%)
>>> - - - - - - - - - - - - - - - - - - - - - - - - -
>>> [ ID][Role] Interval Transfer Bitrate Jitter
>>> Lost/Total Datagrams
>>> [ 5][TX-C] 0.00-10.00 sec 754 MBytes 632 Mbits/sec 0.000 ms
>>> 0/647731 (0%) sender
>>> [ 5][TX-C] 0.00-10.12 sec 105 MBytes 87.2 Mbits/sec 0.245 ms
>>> 571090/647649 (88%) receiver
>>> [ 7][RX-C] 0.00-10.00 sec 1009 MBytes 846 Mbits/sec 0.000 ms
>>> 0/761013 (0%) sender
>>> [ 7][RX-C] 0.00-10.12 sec 86.2 MBytes 71.4 Mbits/sec 0.185 ms
>>> 674392/737139 (91%) receiver
>>>
>>>
>
> Taking a look at this iperf output I recalled that 80% of my setups
> suffered from similar issue after transitioning from 1{1,2}-STABLE to
> 13-STABLE about a mounth ago. I have asked on IRC but nobody confirmed
> similar problems so I have reduced MTU to 8900 from the original 9000 on
> some vlan(4) interfaces to solve the issue. I am using mostly vlan(4)s
> over LACP lagg(4)s created on NICs. So far (for FreeBSD 11 and 12)
> setting MTU 9000 on physical NIC was sufficient to make it work, now I
> have set MTU 9000 on NICs and reduced MTU 8900 on vlan(4)s.
TCP announces an MSS of 1440, which corresponds to an MTU of 1500 byte.
So I guess your problem is different.
Is your physical MTU 9000 bytes?
Best regards
Michael
>
>>>
>>>
>>> On Tuesday, March 16, 2021 4:16:15 AM EDT tue...@freebsd.org wrote:
>>>>> On 15. Mar 2021, at 12:56, Blake Hartshorn <cont...@blakehartshorn.com>
>>>>> wrote:
>>>>>
>>>>> The short version, when I use FreeBSD 13, delivering data can take 5
>>>>> minutes for 1MB over SSH or HTTP when using IPv6. This problem does not
>>>>> happen with IPv4. I installed FreeBSD 12 and Linux on that same device,
>>>>> neither had the problem.
>>>>>
>>>>> Did some troubleshooting with Linode, have ultimately ruled the network
>>>>> itself out at this point. When the server is on FreeBSD 13, it can
>>>>> download quickly over IPv6, but not deliver. Started investigating after
>>>>> noticing my SSH session was lagging when cat'ing large files or running
>>>>> builds. This problem even occurs between VMs in the same datacenter. I
>>>>> generated a 1MB file of base64 garbage served by nginx for testing. IPv6
>>>>> is being configured by SLAAC and on both 12 and 13 installs was setup by
>>>>> the installer. Linode uses Linux/KVM hosts for their virtual machines so
>>>>> it's running on that virtual adapter.
>>>>>
>>>>> I asked on the forums, another user recommended going to the mailing
>>>>> lists instead. Does anyone know if config settings need to be different
>>>>> on 13? Did I maybe just find a real issue? I can provide any requested
>>>>> details. Thanks!
>>>> Could you prove a .pcap tracefile, one from the sender, one from the
>>>> recevier, of
>>>> a TCP/IPv6 connection, which doesn't work as expected. For example, use
>>>> your 1MB
>>>> base64 garbage transfer.
>>>>
>>>> Best regardes
>>>> Michael
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> freebsd-net@freebsd.org mailing list
>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>>>
>>>>
>>> <freebsd12-client.pcap.gz><freebsd13-server.pcap.gz>
>>
>> _______________________________________________
>> freebsd-net@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
>>
>
>
> --
> Marek Zarychta
>
_______________________________________________
freebsd-net@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
