I tested single IPv6 route performance with two 100G NICs.
The test tool is Trex which can send packets 100Gb/s easily.
But VPP(1908.1) has a lot of tx-errors while forwarding the packets.
20 threads, 20 rx queues, 20 tx queues, 60G Hugepages.
When T-rex send 100Gb/s, VPP droped nearly 10% packets.
How can I get rid of the tx-error? Is it possible to improve the performance?
When I set 30 workers or 10 workers with same tx/rx queues, the tx-error is not
changed.
Information
Trex's result:
Send 8.2Mpps, Recv 7.33Mpps
Global Statistitcs
connection : localhost, Port 4501 total_tx_L2 : 98.71 Gb/sec
version : STL @ v2.81 total_tx_L1 : 100.02
Gb/sec
cpu_util. : 5.67% @ 14 cores (14 per dual port) total_rx : 88.25
Gb/sec
rx_cpu_util. : 0.0% / 0 pkt/sec total_pps : 8.2 Mpkt/sec
async_util. : 0.16% / 1.32 KB/sec drop_rate : 10.46 Gb/sec
total_cps. : 0 cps/sec queue_full : 0 pkts
Port Statistics
port | 0 | 1 | total
-----------+-------------------+-------------------+------------------
owner | root | root |
link | UP | UP |
state | TRANSMITTING | IDLE |
speed | 100 Gb/s | 100 Gb/s |
CPU util. | 5.67% | 0.0% |
-- | | |
Tx bps L2 | 98.71 Gbps | 0 bps | 98.71 Gbps
Tx bps L1 | 100.02 Gbps | 0 bps | 100.02 Gbps
Tx pps | 8.2 Mpps | 0 pps | 8.2 Mpps
Line Util. | 100.02 % | 0 % |
--- | | |
Rx bps | 0 bps | 88.25 Gbps | 88.25 Gbps
Rx pps | 0 pps | 7.33 Mpps | 7.33 Mpps
---- | | |
opackets | 223589643 | 0 | 223589643
ipackets | 2 | 199741988 | 199741990
obytes | 336278835072 | 0 | 336278835072
ibytes | 180 | 300411956124 | 300411956304
tx-pkts | 223.59 Mpkts | 0 pkts | 223.59 Mpkts
rx-pkts | 2 pkts | 199.74 Mpkts | 199.74 Mpkts
tx-bytes | 336.28 GB | 0 B | 336.28 GB
rx-bytes | 180 B | 300.41 GB | 30
vpp# show int
Name Idx State MTU (L3/IP4/IP6/MPLS) Counter
Count
HundredGigabitEthernet82/0/0 1 up 9000/0/0/0 rx packets
5038457978
rx bytes 7557686967000
tx packets 8
tx bytes 752
ip6 5038458089
rx-miss 29522
HundredGigabitEthernet83/0/0 2 up 9000/0/0/0 tx packets
5038458485
tx bytes 7557687717666
tx-error 526772148
local0 0 down 0/0/0/0
vpp# show err
Count Node Reason
14 ip6-icmp-input router advertisements sent
32038454 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32036988 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32038478 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32027733 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32038748 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32042218 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32038604 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32029906 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32038974 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32041972 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32038119 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
32032684 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
21625413 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
21627658 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
21627926 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
21610610 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
21618128 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
21614224 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
21615517 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
21617616 HundredGigabitEthernet83/0/0-tx Tx packet drops (dpdk tx
failure)
vpp# show int addr
HundredGigabitEthernet82/0/0 (up):
L3 2d01:1111::1/64
HundredGigabitEthernet83/0/0 (up):
L3 2d02:1111::1/64
local0 (dn):
vpp#
vpp# show ip6 fib
ipv6-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ]
locks:[src:plugin-hi:1, src:recursive-resolution:1, src:default-route:1, ]
::/0
unicast-ip6-chain
[@0]: dpo-load-balance: [proto:ip6 index:6 buckets:1 uRPF:15
to:[6015241112:8938648292432]]
[0] [@19]: dpo-load-balance: [proto:ip6 index:19 buckets:1 uRPF:14 to:[0:0]
via:[6015241112:8938648292432]]
[0] [@5]: ipv6 via 2d02:1111::100 HundredGigabitEthernet83/0/0: mtu:9000
0c42a150d9860c42a150d98286dd
Hugepages:
cat /proc/meminfo | grep Hu
AnonHugePages: 114688 kB
HugePages_Total: 60
HugePages_Free: 54
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB
startup.conf
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
gid vpp
}
api-trace {
## This stanza controls binary API tracing. Unless there is a very strong
reason,
## please leave this feature enabled.
on
## Additional parameters:
##
## To set the number of binary API trace records in the circular buffer,
configure nitems
##
## nitems <nnn>
##
## To save the api message table decode tables, configure a filename. Results
in /tmp/<filename>
## Very handy for understanding api message changes between versions,
identifying missing
## plugins, and so forth.
##
## save-api-table <filename>
}
api-segment {
gid vpp
}
socksvr {
default
}
cpu {
## In the VPP there is one main thread and optionally the user can create
worker(s)
## The main thread and worker thread(s) can be pinned to CPU core(s) manually
or automatically
## Manual pinning of thread(s) to CPU core(s)
## Set logical CPU core where main thread runs, if main core is not set
## VPP will use core 1 if available
#main-core 3
## Set logical CPU core(s) where worker threads are running
#corelist-workers 5,7,9,11,13,15,17,19,21,23
## Automatic pinning of thread(s) to CPU core(s)
## Sets number of CPU core(s) to be skipped (1 ... N-1)
## Skipped CPU core(s) are not used for pinning main thread and working
thread(s).
## The main thread is automatically pinned to the first available CPU core and
worker(s)
## are pinned to next free CPU core(s) after core assigned to main thread
skip-cores 2
## Specify a number of workers to be created
## Workers are pinned to N consecutive CPU cores while skipping "skip-cores"
CPU core(s)
## and main thread's CPU core
workers 20
## Set scheduling policy and priority of main and worker threads
## Scheduling policy options are: other (SCHED_OTHER), batch (SCHED_BATCH)
## idle (SCHED_IDLE), fifo (SCHED_FIFO), rr (SCHED_RR)
# scheduler-policy fifo
## Scheduling priority is used only for "real-time policies (fifo and rr),
## and has to be in the range of priorities supported for a particular policy
# scheduler-priority 50
}
buffers {
## Increase number of buffers allocated, needed only in scenarios with
## large number of interfaces and worker threads. Value is per numa node.
## Default is 16384 (8192 if running unpriviledged)
buffers-per-numa 512000
## Size of buffer data area
## Default is 2048
# default data-size 2048
}
dpdk {
## Change default settings for all interfaces
dev default {
## Number of receive queues, enables RSS
## Default is 1
num-rx-queues 20
## Number of transmit queues, Default is equal
## to number of worker threads or 1 if no workers treads
num-tx-queues 20
## Number of descriptors in transmit and receive rings
## increasing or reducing number can impact performance
## Default is 1024 for both rx and tx
#num-rx-desc 2048
#num-tx-desc 2048
## VLAN strip offload mode for interface
## Default is off
# vlan-strip-offload on
}
## Whitelist specific interface by specifying PCI address
dev 0000:82:00.0
dev 0000:83:00.0
#vdev
eth_bond0,mode=2,slave=0000:02:00.0,slave=0000:03:00.0,slave=0000:81:00.0,xmit_policy=l34
#vdev
eth_bond1,mode=2,slave=0000:02:00.1,slave=0000:03:00.1,slave=0000:81:00.1,xmit_policy=l34
## Blacklist specific device type by specifying PCI vendor:device
## Whitelist entries take precedence
# blacklist 8086:10fb
## Set interface name
# dev 0000:02:00.1 {
# name eth0
# }
## Whitelist specific interface by specifying PCI address and in
## addition specify custom parameters for this interface
# dev 0000:02:00.1 {
# num-rx-queues 2
# }
## Change UIO driver used by VPP, Options are: igb_uio, vfio-pci,
## uio_pci_generic or auto (default)
# uio-driver vfio-pci
## Disable multi-segment buffers, improves performance but
## disables Jumbo MTU support
# no-multi-seg
## Change hugepages allocation per-socket, needed only if there is need for
## larger number of mbufs. Default is 256M on each detected CPU socket
socket-mem 2048,2048
## Disables UDP / TCP TX checksum offload. Typically needed for use
## faster vector PMDs (together with no-multi-seg)
# no-tx-checksum-offload
}
NICs:
Mellanox ConnectX-5 MCX555A-ECAT single port. x 2
[root@localhost v2.81]# ./dpdk_nic_bind.py -s
Network devices using DPDK-compatible driver
============================================
0000:82:00.0 'MT27800 Family [ConnectX-5]' drv=mlx5_core unused=uio_pci_generic
0000:83:00.0 'MT27800 Family [ConnectX-5]' drv=mlx5_core unused=uio_pci_generic
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16929): https://lists.fd.io/g/vpp-dev/message/16929
Mute This Topic: https://lists.fd.io/mt/75416173/21656
Mute #vpp: https://lists.fd.io/g/fdio+vpp-dev/mutehashtag/vpp
Mute #dpdk: https://lists.fd.io/g/fdio+vpp-dev/mutehashtag/dpdk
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
