On 6/13/22 20:25, Mike Jakubik wrote:
Hello,
I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s,
however, I am unable to get much more than 6Gb/s when testing with iperf3.
The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM,
Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter)
They are connected to a Dell N3224PX-ON switch. Both servers are idle and not
in use, with a fresh install of stable/13-ebea872f8, nothing running on them
except ssh, sendmail, etc.
When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I
have tried just about every knob listed in
https://calomel.org/freebsd_network_tuning.html with little impact on the
performance. The network cards have HW LRO enabled as per the driver
documentation (though this only seems to lower IRQ usage with no impact on
actual throughput).
The same exact servers tested on Linux (fedora 34) produced nearly 3x the
performance (see attached screenshots), i was able to get a steady 14.6Gb/s
rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to
avg at around 6Gbs but it is very sporadic during the iperf run.
I have run out of ideas, any suggestions are welcome. Considering Netflix uses
very similar HW and they push 400 Gb/s tells me there is something really wrong
here or Netflix isnt sharing all their secret sauce.
Some ideas:
Try to disable "rxpause,txpause" when setting the media.
Keep HW LRO off for now, it doesn't work for large number of connections.
What is the CPU usage during test? Is iperf3 running on a CPU-core which
has direct access to the NIC's numa domain?
Is the NIC installed in the "correct" PCI high-performance slot?
There are some sysctl knobs which may tell where the problem is, if it's
PCI backpressure or something else.
sysctl -a | grep diag_pci_enable
sysctl -a | grep diag_general_enable
Set these two to 1, then run some traffic and dump all mce sysctls:
sysctl -a | grep mce > dump.txt
--HPS
