Hi all,
While performance testing vpp stable/2206 using TRex, I am seeing a
huge drop in performance when nat44 is enabled. I was seeing 14Mbps of
packet drop when TRex was pushing at 20kpps on an Xeon(R) Gold 6342
CPU with E-810 100G NIC.
During the run I could see congestion drops in
nat44-in2out-worker-handoff. Could this be the way I have configured
vpp? Because this is obviously not the limits of VPP on this hardware.
Any clues to figure out what's happening would be great. If needed I
can run tests again and change the configs.
Configuration summery is as follows, full config is attached:
* Intel(R) Xeon(R) Gold 6342 CPU @ 2.80GHz
* Two E-810 100G NIC
* One worker (and hence 1 rx queue) per nic
* Main core and workers in same numa node as NIC
* 5 workers with two dedicated for NAT
* /30 network for each iface and static routes
NAT commands:
set nat workers 2-3
nat44 plugin enable sessions 12800000
nat mss-clamping 1452
nat44 add address 100.64.0.1-100.64.1.254
nat44 add address 100.64.2.1-100.64.3.254
nat44 add address 100.64.4.1-100.64.5.254
nat44 add address 100.64.6.1-100.64.7.254
set interface nat44 in HundredGigabitEthernetb4/0/0
set interface nat44 out HundredGigabitEthernetb1/0/0
Observations while running the tests:
vpp# sh threads
ID Name Type LWP Sched Policy (Priority)
lcore Core Socket State
0 vpp_main 7190 other (0)
25 1 1
1 vpp_wk_0 workers 7195 other (0)
26 2 1
2 vpp_wk_1 workers 7196 other (0)
27 3 1
3 vpp_wk_2 workers 7197 other (0)
28 4 1
4 vpp_wk_3 workers 7198 other (0)
29 5 1
5 vpp_wk_4 workers 7199 other (0)
30 6 1
vpp# sh int rx-placement
Thread 1 (vpp_wk_0):
node dpdk-input:
HundredGigabitEthernetb1/0/0 queue 0 (polling)
Thread 2 (vpp_wk_1):
HundredGigabitEthernetb4/0/0 queue 0 (polling)
Thread 5 (vpp_wk_4):
node virtio-input:
tap0 queue 0 (polling)
tap1 queue 0 (polling)
tap2 queue 0 (polling)
tap3 queue 0 (polling)
vpp# sh nat workers
2 workers
vpp_wk_2
vpp_wk_3
vpp# sh errors
Count Node Reason
Severity
139640 nat44-out2in-worker-handoff do handoff
error
1 lldp-input lldp packets received on
disabled i error
5 llc-input unknown llc
ssap/dsap error
62900 nat44-in2out-worker-handoff congestion
drop error
202605 nat44-in2out-worker-handoff do handoff
error
5 llc-input unknown llc
ssap/dsap error
vpp# sh int HundredGigabitEthernetb1/0/0 HundredGigabitEthernetb4/0/0
Name Idx State MTU (L3/IP4/IP6/MPLS)
Counter Count
HundredGigabitEthernetb1/0/0 1 up 1500/0/0/0 rx
packets 481773
rx bytes 81900507
tx packets 481519
tx bytes 140122029
drops 19
ip4 481754
HundredGigabitEthernetb4/0/0 2 up 1500/0/0/0 rx
packets 701806
rx bytes 204222238
tx packets 481531
tx bytes 81860270
drops 20
ip4 701786
vpp# sh nat44 summary
max translations per thread: 12800000 fib 0
udp LRU min session timeout 5697 (now 5729)
udp LRU min session timeout 5697 (now 5729)
total sessions: 546980 (timed out: 428296)
tcp sessions:
total: 0 (timed out: 0)
established: 0 (timed out: 0)
transitory: 0 (timed out: 0)
udp sessions:
total: 546980 (timed out: 428296)
icmp sessions:
total: 0 (timed out: 0)
other sessions:
total: 0 (timed out: 0)
While looking at the node_names following were the top in per node counters
nat44-in2out-worker-handoff
nat44-ed-in2out-slowpath
dpdk-input
virtio-input
ip4-sv-reassembly-feature
ip4-input-no-checksum
ethernet-input
unix-epoll-input
# Example for creating your program by specifying buffers to send, without relaying on pcap file
from trex.astf.api import *
import argparse
# we can send either Python bytes type as below:
http_req = b'GET /3384 HTTP/1.1\r\nHost: 22.0.0.3\r\nConnection: Keep-Alive\r\nUser-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)\r\nAccept: */*\r\nAccept-Language: en-us\r\nAccept-Encoding: gzip, deflate, compress\r\n\r\n'
# or we can send Python string containing ascii chars, as below:
http_response = 'HTTP/1.1 200 OK\r\nServer: Microsoft-IIS/6.0\r\nContent-Type: text/html\r\nContent-Length: 32000\r\n\r\n<html><pre>**********</pre></html>'
class Prof1():
def __init__(self):
pass # tunables
def create_profile(self):
# client commands
prog_c = ASTFProgram(stream=False)
prog_c.send_msg(http_req)
prog_c.recv_msg(1)
prog_s = ASTFProgram(stream=False)
prog_s.recv_msg(1)
prog_s.send_msg(http_response)
# ip generator
ip_gen_c = ASTFIPGenDist(ip_range=["16.0.0.0", "16.0.127.255"], distribution="seq")
ip_gen_s = ASTFIPGenDist(ip_range=["48.0.0.0", "48.0.255.255"], distribution="seq")
ip_gen = ASTFIPGen(glob=ASTFIPGenGlobal(ip_offset="1.0.0.0"),
dist_client=ip_gen_c,
dist_server=ip_gen_s)
# template
temp_c = ASTFTCPClientTemplate(program=prog_c,ip_gen=ip_gen)
temp_s = ASTFTCPServerTemplate(program=prog_s) # using default association
template = ASTFTemplate(client_template=temp_c, server_template=temp_s)
# profile
profile = ASTFProfile(default_ip_gen=ip_gen, templates=template)
return profile
def get_profile(self, tunables, **kwargs):
parser = argparse.ArgumentParser(description='Argparser for {}'.format(os.path.basename(__file__)),
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
args = parser.parse_args(tunables)
return self.create_profile()
def register():
return Prof1()
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
cli-listen /run/vpp/cli.sock
gid vpp
exec /usr/local/config/init.cmd
}
api-trace {
}
api-segment {
gid vpp
}
socksvr {
default
}
memory {
main-heap-size 32G
}
cpu {
main-core 25
corelist-workers 26-30
}
buffers {
buffers-per-numa 128000
}
dpdk {
dev default {
num-rx-desc 2048
num-tx-desc 2048
}
dev 0000:b1:00.0 {
workers 0
}
dev 0000:b4:00.0 {
workers 1
}
}
statseg {
size 1G
per-node-counters on
}
comment {Enable NAT in Endpoint Dependent Mode}
set nat workers 2-3
nat44 plugin enable sessions 12800000
comment { Internet }
set int mtu 1500 HundredGigabitEthernetb1/0/0
set int state HundredGigabitEthernetb1/0/0 up
set int ip address HundredGigabitEthernetb1/0/0 192.168.10.1/30
comment { Access Interfaces }
set int mtu 1500 HundredGigabitEthernetb4/0/0
set int state HundredGigabitEthernetb4/0/0 up
set int ip address HundredGigabitEthernetb4/0/0 192.168.11.1/30
comment { Default route for ipv4 }
ip route add 0.0.0.0/0 via 192.168.10.2 HundredGigabitEthernetb1/0/0
comment { NAT commands }
nat mss-clamping 1452
nat44 add address 100.64.0.1-100.64.1.254
nat44 add address 100.64.2.1-100.64.3.254
nat44 add address 100.64.4.1-100.64.5.254
nat44 add address 100.64.6.1-100.64.7.254
set interface nat44 in HundredGigabitEthernetb4/0/0
set interface nat44 out HundredGigabitEthernetb1/0/0
comment { Timeouts to low to test high connection/sec }
set nat timeout udp 10 tcp-established 1260 tcp-transitory 45 icmp 40
comment { Add PPPOE tap }
create tap host-if-name pppoetap hw-addr 02:fe:29:bf:68:64 rx-ring-size 4096
tx-ring-size 4096
set int state tap0 up
create pppoe cp cp-if-index 3
comment { Add tap for DHCP }
create tap host-if-name lstack host-ip4-addr 192.168.7.1/30 hw-addr
02:fe:21:c3:a6:fd rx-ring-size 4096 tx-ring-size 4096
set int ip address tap1 192.168.7.2/30
set int state tap1 up
comment { Add tap for IPfix }
create tap host-if-name collector host-ip4-addr 192.168.7.5/30 hw-addr
02:fe:dc:b8:dc:8d rx-ring-size 4096 tx-ring-size 4096
set int ip address tap2 192.168.7.6/30
set ipfix exporter collector 192.168.7.5 src 192.168.7.6 template-interval 20
set int state tap2 up
comment { Add tal tap }
create tap host-if-name taltap hw-addr 02:fe:4f:a8:d0:4f rx-ring-size 4096
tx-ring-size 4096
set int state tap3 up
comment { Routes to make Trex work }
ip route add 16.0.0.0/8 via 192.168.11.2
ip route add 48.0.0.0/8 via 192.168.10.2
comment { All tap to last worker }
set int rx-placement tap0 queue 0 worker 4
set int rx-placement tap1 queue 0 worker 4
set int rx-placement tap2 queue 0 worker 4
set int rx-placement tap3 queue 0 worker 4
nat ipfix logging enable
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21997): https://lists.fd.io/g/vpp-dev/message/21997
Mute This Topic: https://lists.fd.io/mt/94256640/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/leave/1480452/21656/631435203/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
