ConnectX-7 performance anomaly

[Kamil Vojanec]

Hello,

while benchmarking the Nvidia ConnectX-7 400Gbe NIC, I encountered some interesting behavior: when processing short packets (72 B), the maximum packet rate I can achieve is 230 Mpps. However, when I increase the packet size to 512 B briefly and then bring the packet size down to 72 B again, the maximum throughput is now 260 Mpps and seems to stabilize at that value. According to the Nvidia performance report (https://fast.dpdk.org/doc/perf/DPDK_24_07_NVIDIA_NIC_performance_report.pdf), the maximum achievable packet rate of ConnectX-7 NICs is 266 Mpps and that is what I am trying to reproduce. I have observed this behavior with dpdk-l3fwd as well as with my own DPDK application.

My setup is as follows:

CPU: AMD EPYC 9554P 64-Core Processor

Application commandline:



Generator: Spirent 400G generator



Kamil Vojanec
CESNET z.s.p.o.

sudo dpdk_example_l3fwd -a0000:05:00.0 -l0-32 --main-lcore=32  -- -p0x1 --config "(0,0,0),(0,1,1),(0,2,2),(0,3,3),(0,4,4),(0,5,5),(0,6,6),(0,7,7),(0,8,8),(0,9,9),(0,10,10),(0,11,11),(0,12,12),(0,13,13),(0,14,14),(0,15,15),(0,16,16),(0,17,17),(0,18,18),(0,19,19),(0,20,20),(0,21,21),(0,22,22),(0,23,23),(0,24,24),(0,25,25),(0,26,26),(0,27,27),(0,28,28),(0,29,29),(0,30,30),(0,31,31)" --eth-dest=0,00:10:94:00:00:02 Would anyone be able to share some insight as to what might be happening here?

smime.p7s
Description: S/MIME Cryptographic Signature