Hi VPP team,
need to wrap my head around why VPP is not even able to process 10 Mpps one single-core setup.
Afaik VPP xconnect ( comparable to l2fwd) or L3-routing ( l3wfd) should yield 10 Mpps even back in 2017 ( slide 15 https://wiki.fd.io/images/3/31/Benchmarking-sw-data-planes-Dec5_2017.pdf or p16 https://lfnetworking.org/wp-content/uploads/sites/7/2022/06/benchmarking_sw_data_planes_skx_bdx_mar07_2019.pdf )
My setup is quite up-to-date:
Traffic Gen: TRex running on Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz
DuT: VPP 22.10 running on Icelake Intel(R) Xeon(R) Platinum 8352Y CPU @ 2.20GHz
NICs: ConnectX-6 Dx Dualport
OS: Ubuntu Server 20.04
VPP compiled from src . Tag 22.10. Incl. changes to compile for MLX5_PMD.
VPP show dpdk version:
DPDK Version: DPDK 22.07.0
DPDK EAL init args: --in-memory --no-telemetry --file-prefix vpp -a 0000:4b:00.0,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1 -a 0000:4b:00.1,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1
DPDK EAL init args: --in-memory --no-telemetry --file-prefix vpp -a 0000:4b:00.0,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1 -a 0000:4b:00.1,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1
QUESTION:
Running the a plain DPDK 22.07 example l2fwd or l3fwd yields > 10 Mpps ( 100 Gbps line rate) with 1 lcore. Also no additional dpdk options required.
Running the a plain DPDK 22.07 example l2fwd or l3fwd yields > 10 Mpps ( 100 Gbps line rate) with 1 lcore. Also no additional dpdk options required.
Once I run either xconnect or L3 routing with VPP, the same test drops to 6 Mpps or less. It is puzzling why VPP is not even close the same exact dpdk l2fwd/ l3wfd example? Any advise?
Here are the additional cmds to configure either l2fwd or L3-routing in VPP:
```bash
set interface l2 xconnect HundredGigabitEthernet4b/0/0 HundredGigabitEthernet4b/0/1
set interface l2 xconnect HundredGigabitEthernet4b/0/1 HundredGigabitEthernet4b/0/0
set interface l2 xconnect HundredGigabitEthernet4b/0/1 HundredGigabitEthernet4b/0/0
# l3 routing
set interface ip address HundredGigabitEthernet4b/0/0 10.10.1.1/24
set interface ip address HundredGigabitEthernet4b/0/1 10.10.2.1/24
ip route add 16.0.0.0/8 via 10.10.1.2
ip route add 48.0.0.0/8 via 10.10.2.2
set interface ip address HundredGigabitEthernet4b/0/1 10.10.2.1/24
ip route add 16.0.0.0/8 via 10.10.1.2
ip route add 48.0.0.0/8 via 10.10.2.2
```
Thread 0 vpp_main (lcore 4)
Time 2472.6, 10 sec internal node vector rate 0.00 loops/sec 1244718.44
vector rates in 0.0000e0, out 8.0885e-4, drop 0.0000e0, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
HundredGigabitEthernet4b/0/0-o active 1 1 0 1.67e4 1.00
HundredGigabitEthernet4b/0/0-t active 1 1 0 1.81e4 1.00
HundredGigabitEthernet4b/0/1-o active 1 1 0 2.53e3 1.00
HundredGigabitEthernet4b/0/1-t active 1 1 0 2.43e3 1.00
acl-plugin-fa-cleaner-process event wait 0 0 1 7.26e3 0.00
admin-up-down-process event wait 0 0 1 8.95e3 0.00
api-rx-from-ring any wait 0 0 130 5.58e4 0.00
avf-process event wait 0 0 1 5.74e3 0.00
bfd-process event wait 0 0 1 3.13e3 0.00
bond-process event wait 0 0 1 2.27e3 0.00
cnat-scanner-process event wait 0 0 1 5.74e3 0.00
dhcp-client-process any wait 0 0 3 1.10e4 0.00
dhcp6-client-cp-process any wait 0 0 1 3.22e3 0.00
dhcp6-pd-client-cp-process any wait 0 0 1 2.83e3 0.00
dhcp6-pd-reply-publisher-proce event wait 0 0 1 3.21e3 0.00
dhcp6-reply-publisher-process event wait 0 0 1 3.01e3 0.00
dpdk-process any wait 0 0 824 1.18e7 0.00
fib-walk any wait 0 0 1235 2.86e3 0.00
flow-report-process any wait 0 0 1 3.32e3 0.00
flowprobe-timer-process any wait 0 0 1 6.21e3 0.00
igmp-timer-process event wait 0 0 1 5.61e3 0.00
ikev2-manager-process event wait 0 0 1 8.69e3 0.00
ioam-export-process any wait 0 0 1 4.40e3 0.00
ip-neighbor-event event wait 0 0 1 2.77e3 0.00
ip4-full-reassembly-expire-wal any wait 0 0 49356 1.66e3 0.00
ip4-neighbor-age-process event wait 0 0 1 2.66e3 0.00
ip4-sv-reassembly-expire-walk any wait 0 0 247 2.25e3 0.00
ip6-full-reassembly-expire-wal any wait 0 0 49356 1.78e3 0.00
ip6-mld-process any wait 0 0 2470 1.39e3 0.00
ip6-neighbor-age-process event wait 0 0 1 2.89e3 0.00
ip6-ra-process any wait 0 0 2470 1.41e3 0.00
ip6-rs-process any wait 0 0 1 2.86e3 0.00
ip6-sv-reassembly-expire-walk any wait 0 0 247 2.63e3 0.00
l2-arp-term-publisher event wait 0 0 1 2.29e3 0.00
l2fib-mac-age-scanner-process event wait 0 0 1 4.13e3 0.00
lldp-process event wait 0 0 1 1.16e6 0.00
memif-process event wait 0 0 1 6.91e3 0.00
nat44-ei-ha-process event wait 0 0 1 5.65e3 0.00
nsh-md2-ioam-export-process any wait 0 0 1 5.69e3 0.00
rd-cp-process any wait 0 0 1 7.79e3 0.00
send-dhcp6-client-message-proc any wait 0 0 1 3.34e3 0.00
send-dhcp6-pd-client-message-p any wait 0 0 1 7.39e3 0.00
startup-config-process done 1 0 1 1.49e9 0.00
statseg-collector-process time wait 0 0 247 1.07e6 0.00
udp-ping-process any wait 0 0 1 7.07e3 0.00
unix-cli-local:4 active 4 0 9 3.26e12 0.00
unix-cli-new-session any wait 0 0 14 6.39e7 0.00
unix-epoll-input polling 239020776 0 0 1.63e4 0.00
vhost-user-process any wait 0 0 1 8.68e3 0.00
vhost-user-send-interrupt-proc any wait 0 0 1 3.98e3 0.00
vpe-link-state-process event wait 0 0 2 4.06e3 0.00
vrrp-periodic-process event wait 0 0 1 5.51e3 0.00
vxlan-gpe-ioam-export-process any wait 0 0 1 3.04e3 0.00
wg-timer-manager event wait 0 0 1 1.52e5 0.00
---------------
Thread 1 vpp_wk_0 (lcore 6)
Time 2472.6, 10 sec internal node vector rate 0.00 loops/sec 9492929.41
vector rates in 6.5792e6, out 6.5792e6, drop 4.6913e-2, punt 0.0000e0
Name State Calls Vectors Suspends Clocks Vectors/Call
HundredGigabitEthernet4b/0/0-o active 692990734 8631229901 0 1.04e1 12.46
HundredGigabitEthernet4b/0/0-t active 692990734 8631229901 0 4.04e1 12.46
HundredGigabitEthernet4b/0/1-o active 695352295 7636739975 0 1.17e1 10.98
HundredGigabitEthernet4b/0/1-t active 695352295 7636739975 0 3.39e1 10.98
arp-input active 28 30 0 2.03e3 1.07
arp-reply active 28 30 0 1.47e4 1.07
dpdk-input polling 5329333974 16267969992 0 9.33e1 3.05
drop active 105 116 0 1.14e3 1.10
error-drop active 105 116 0 6.06e2 1.10
ethernet-input active 1388343054 16267969992 0 2.27e1 11.72
interface-output active 2 2 0 1.51e3 1.00
ip4-input-no-checksum active 924091013 16267969874 0 1.77e1 17.60
ip4-load-balance active 924091013 16267969874 0 1.02e1 17.60
ip4-lookup active 924091013 16267969874 0 1.46e1 17.60
ip4-rewrite active 924091013 16267969874 0 1.66e1 17.60
lldp-input active 77 88 0 3.54e3 1.14
unix-epoll-input polling 5199376 0 0 2.77e2 0.00
DPDK Version: DPDK 22.07.0
DPDK EAL init args: --in-memory --no-telemetry --file-prefix vpp -a 0000:4b:00.0,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1 -a 0000:4b:00.1,mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1
Name Idx Link Hardware
HundredGigabitEthernet4b/0/0 1 up HundredGigabitEthernet4b/0/0
Link speed: 100 Gbps
RX Queues:
queue thread mode
0 vpp_wk_0 (1) polling
TX Queues:
TX Hash: [name: hash-eth-l34 priority: 50 description: Hash ethernet L34 headers]
queue shared thread(s)
0 no 0
1 no 1
Ethernet address b8:ce:f6:dc:c1:f0
Mellanox ConnectX-4 Family
carrier up full duplex max-frame-size 9022 promisc
flags: admin-up promisc maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
Devargs: mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1
rx: queues 1 (max 1024), desc 4096 (min 0 max 65535 align 1)
tx: queues 2 (max 1024), desc 4096 (min 0 max 65535 align 1)
pci: device 15b3:101d subsystem 15b3:0058 address 0000:4b:00.00 numa 0
switch info: name 0000:4b:00.0 domain id 0 port id 65535
max rx packet len: 65536
promiscuous: unicast on all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter
scatter timestamp rss-hash
rx offload active: ipv4-cksum scatter
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
outer-ipv4-cksum vxlan-tnl-tso gre-tnl-tso geneve-tnl-tso
multi-segs udp-tnl-tso ip-tnl-tso
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
ipv6-ex ipv6 esp l4-dst-only l4-src-only l3-dst-only
l3-src-only
rss active: none
tx burst mode: Enhanced MPW + MULTI + TSO + SWP + CSUM + METADATA
tx burst function: (not available)
rx burst mode: Scalar
rx burst function: (not available)
tx frames ok 8631229902
tx bytes ok 11920117654763
rx frames ok 7636740042
rx bytes ok 2283761417707
rx missed 86941
extended stats:
rx_good_packets 7636740042
tx_good_packets 8631229902
rx_good_bytes 2283761417707
tx_good_bytes 11920117654763
rx_missed_errors 86941
rx_q0_packets 7636740042
rx_q0_bytes 2283761417707
tx_q0_packets 1
tx_q0_bytes 42
tx_q1_packets 8631229901
tx_q1_bytes 11920117654721
rx_unicast_packets 7636826915
rx_unicast_bytes 2314334535080
tx_unicast_packets 8631229901
tx_unicast_bytes 11920117654721
rx_multicast_packets 50
rx_multicast_bytes 4400
rx_broadcast_packets 18
rx_broadcast_bytes 1152
tx_broadcast_packets 1
tx_broadcast_bytes 60
tx_phy_packets 8631229952
rx_phy_packets 7636826983
tx_phy_bytes 11954642578789
rx_phy_bytes 2314334540632
rx_out_of_buffer 86941
HundredGigabitEthernet4b/0/1 2 up HundredGigabitEthernet4b/0/1
Link speed: 100 Gbps
RX Queues:
queue thread mode
0 vpp_wk_0 (1) polling
TX Queues:
TX Hash: [name: hash-eth-l34 priority: 50 description: Hash ethernet L34 headers]
queue shared thread(s)
0 no 0
1 no 1
Ethernet address b8:ce:f6:dc:c1:f1
Mellanox ConnectX-4 Family
carrier up full duplex max-frame-size 9022 promisc
flags: admin-up promisc maybe-multiseg tx-offload intel-phdr-cksum rx-ip4-cksum
Devargs: mprq_en=1,rxqs_min_mprq=1,mprq_log_stride_num=9,txq_inline_mpw=128,rxq_pkt_pad_en=1
rx: queues 1 (max 1024), desc 4096 (min 0 max 65535 align 1)
tx: queues 2 (max 1024), desc 4096 (min 0 max 65535 align 1)
pci: device 15b3:101d subsystem 15b3:0058 address 0000:4b:00.01 numa 0
switch info: name 0000:4b:00.1 domain id 1 port id 65535
max rx packet len: 65536
promiscuous: unicast on all-multicast on
vlan offload: strip off filter off qinq off
rx offload avail: vlan-strip ipv4-cksum udp-cksum tcp-cksum vlan-filter
scatter timestamp rss-hash
rx offload active: ipv4-cksum scatter
tx offload avail: vlan-insert ipv4-cksum udp-cksum tcp-cksum tcp-tso
outer-ipv4-cksum vxlan-tnl-tso gre-tnl-tso geneve-tnl-tso
multi-segs udp-tnl-tso ip-tnl-tso
tx offload active: ipv4-cksum udp-cksum tcp-cksum multi-segs
rss avail: ipv4-frag ipv4-tcp ipv4-udp ipv4-other ipv4 ipv6-tcp-ex
ipv6-udp-ex ipv6-frag ipv6-tcp ipv6-udp ipv6-other
ipv6-ex ipv6 esp l4-dst-only l4-src-only l3-dst-only
l3-src-only
rss active: none
tx burst mode: Enhanced MPW + MULTI + TSO + SWP + CSUM + METADATA
tx burst function: (not available)
rx burst mode: Scalar
rx burst function: (not available)
tx frames ok 7636739976
tx bytes ok 2283761412529
rx frames ok 8631229952
rx bytes ok 11920117658741
rx missed 155395
extended stats:
rx_good_packets 8631229952
tx_good_packets 7636739976
rx_good_bytes 11920117658741
tx_good_bytes 2283761412529
rx_missed_errors 155395
rx_q0_packets 8631229952
rx_q0_bytes 11920117658741
tx_q0_packets 1
tx_q0_bytes 42
tx_q1_packets 7636739975
tx_q1_bytes 2283761412487
rx_unicast_packets 8631385295
rx_unicast_bytes 11954858334100
tx_unicast_packets 7636739975
tx_unicast_bytes 2283761412487
rx_multicast_packets 40
rx_multicast_bytes 3520
rx_broadcast_packets 12
rx_broadcast_bytes 768
tx_broadcast_packets 1
tx_broadcast_bytes 60
tx_phy_packets 7651773752
rx_phy_packets 14044813088
tx_phy_bytes 2315270535363
rx_phy_bytes 19454244496723
rx_phy_discard_packets 5413427741
rx_out_of_buffer 155395
local0 0 down local0
Link speed: unknown
local
startup.conf
Description: Binary data
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22396): https://lists.fd.io/g/vpp-dev/message/22396 Mute This Topic: https://lists.fd.io/mt/95959719/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
