Hello, I am facing a problem using VLAN tag rewrite functionality on VPP.
The communication between a kernel vlan interface and a VPP VLAN sub-interface
fails. The ICMP is working but TCP/UDP packets are dropped by the kernel (just
one side - see the test description)
Find attached the detailed topology, test details, the reproduction
instructions, vpp version and 'show' commands output and tcpdump capture files
as well.
Please advise me with the way forward:
- It can be a configuration problem - what should be changed?
- Might be a fault in kernel checksum (has a low chance, but can happen)
- Can be a fault in VPP tag rewrite - can I raise a ticket? Do you need
additional information?
1. Network Topology
See attached file: topo.txt
2. Test Description
The 'Host A' should reach the 'Host B' by eth1.100 and respectively veth1
interfaces. The 'ping' works, but when trying to establish a TCP connection
(ncat/iperf) the connection fails.
2/A Client is on 'Host A'
Based on captured trace the SYN arrives to veth1 but it is dropped and does
not reach the server listening on 'host B', 'orange' nw namespace.
2/B Client is on 'Host B', 'orange' namespace
Based on trace the SYN is received and replied by the server on 'Host A'.
The SYN ACK can be seen in tcpdump, but dropped by the kernel.
Seems that the case when VPP adding VLAN tag to the packet is working but when
removing the VLAN tag from the packet the kernel drops the packet.
Based on kernel statistics the TcpInCsumErrors counter increasing, however the
VLAN is on ethernet level.
3. Reproduction Steps
See attached file: test-vtr-tcpfailure.txt
4. VPP Version and 'show' Command Outputs
See attached file: test-vtr-cmds.txt
5. Capture Files
Tcpdump captures:
ext-in-tcp-failure-sut-eth1.pcap
ext-in-tcp-failure-sut-veth1.pcap
ext-in-tcp-failure-tester.pcap
out-tcp-failure-sut-eth1.pcap
out-tcp-failure-sut-veth1.pcap
out-tcp-failure-tester.pcap
--
Laszlo Kiraly
+---------------+
| host-eth0.100 |------x l2 xconnect
| POP 1 | |
+---------------+ +------------+
| host-eth0 | | host-veth0 |
+---------------+ +------------+ VPP
_______________________________________________________________________________
+----------+ 172.10.0.253 +---------+ +-------+ +-------+ kernel
| eth1.100 |-------------------| eth0 | | veth0 |----| veth1 |
+----------+ +---------+ +-------+ +-------+
| eth1 | 172.10.0.1
+----------+
Host A Host B (VPP L2 pipe and VLAN tag rewrite)
eth1.100 kernel vlan i/f eth0 kernel interface
veth0, veth1 veth interfaces
veth1 in 'orange' network namespace
# Create the network
docker network create bridge-2
# Create and set up the rvm-tester container
cat Dockerfile << EOF
FROM networkstatic/iperf3
RUN apt-get update \
&& apt-get install -y tcpdump ncat telnet procps psmisc\
&& rm -rf /var/lib/apt/lists/*
ENTRYPOINT [ "tail", "-f", "/dev/null" ]
EOF
docker pull networkstatic/iperf3
docker build . -tag rvm-tester:latest
rm Dockerfile
docker run --cap-add=NET_ADMIN --rm -d --network bridge-2 -v
/tmp/tcpdump:/tmp/tcpdump --name rvm-tester rvm-tester:latest tail -f /dev/null
docker exec rvm-tester ip link set eth0 down
docker exec rvm-tester ip link add link eth0 name eth0.100 type vlan id 100
docker exec rvm-tester ip link set eth0 up
docker exec rvm-tester ip addr add 172.10.0.253/24 dev eth0.100
# Get the image and run vpp in docker container
docker pull ghcr.io/edwarnicke/govpp/vpp:v22.02-rc0-165-gb44d0defb
docker run --privileged --rm -d --network bridge-2 -v /tmp/tcpdump:/tmp/tcpdump
--name vpp-sut ghcr.io/edwarnicke/govpp/vpp:v22.02-rc0-165-gb44d0defb tail -f
/dev/null
# Create vpp config
cat > vpp.conf << EOF
unix {
nodaemon
log /var/log/vpp/vpp.log
full-coredump
cli-listen /var/run/vpp/cli.sock
gid vpp
}
buffers {
buffers-per-numa 32768
default data-size 3776
}
api-trace {
on
}
api-segment {
gid vpp
}
socksvr {
socket-name /var/run/vpp/api.sock
}
statseg {
socket-name /var/run/vpp/stats.sock
}
cpu {
}
plugins {
plugin dpdk_plugin.so { disable }
}
EOF
# Copy vpp config to container
docker cp vpp.conf vpp-sut:/etc/vpp/vpp.conf
rm vpp.conf
# install additional tools
docker exec vpp-sut apt update
docker exec vpp-sut apt install -y ncat jq
# Set the network inside the container
docker exec vpp-sut ip netns add orange
docker exec vpp-sut ip link add veth0 type veth peer name veth1
docker exec vpp-sut ip addr add 192.168.111.1/24 dev veth0
docker exec vpp-sut ip link set veth0 up
docker exec vpp-sut ip link set veth1 netns orange
docker exec vpp-sut ip netns exec orange ip addr add 172.10.0.1/24 dev veth1
MAC=$(docker exec vpp-sut ip -j a s dev eth0 | jq '.[0].address' --)
docker exec vpp-sut ip netns exec orange ip link set veth1 addr ${MAC//\"}
docker exec vpp-sut ip netns exec orange ip link set veth1 up
docker exec vpp-sut ip link set eth0 promisc on
# Set up vpp
docker exec vpp-sut mkdir -p /var/log/vpp
docker exec vpp-sut vpp -c /etc/vpp/vpp.conf &
docker exec vpp-sut vppctl create host-interface name eth0
docker exec vpp-sut vppctl create host-interface name veth0
docker exec vpp-sut vppctl set acl-plugin acl permit src ::/0 dst ::/0 proto 58
sport 0 dport 134, permit src ::/0 dst ::/0 proto 58 sport 0 dport 136, deny
src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0, deny src ::/0 dst ::/0
proto 0 sport 0 dport 0
docker exec vpp-sut vppctl set acl-plugin acl permit src 0.0.0.0/0 dst
0.0.0.0/0 proto 58 sport 133 dport 0, permit src ::/0 dst ::/0 proto 58 sport
135 dport 0, deny src 0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0, deny src
0.0.0.0/0 dst 0.0.0.0/0 proto 0 sport 0 dport 0
docker exec vpp-sut vppctl set acl-plugin interface host-eth0 input acl 0
docker exec vpp-sut vppctl set acl-plugin interface host-eth0 output acl 1
docker exec vpp-sut vppctl set interface rx-mode host-veth0 queue 0 adaptive
docker exec vpp-sut vppctl set interface state host-veth0 up
docker exec vpp-sut vppctl set interface state host-eth0 up
docker exec vpp-sut vppctl create sub-interfaces host-eth0 100
docker exec vpp-sut vppctl set interface l2 tag-rewrite host-eth0.100 pop 1
docker exec vpp-sut vppctl set interface l2 xconnect host-eth0.100 host-veth0
docker exec vpp-sut vppctl set interface l2 xconnect host-veth0 host-eth0.100
docker exec vpp-sut vppctl set interface state host-eth0.100 up
# Check the connection with ping
docker exec rvm-tester ping 172.10.0.1 -c2
PING 172.10.0.1 (172.10.0.1) 56(84) bytes of data.
64 bytes from 172.10.0.1: icmp_seq=1 ttl=64 time=0.171 ms
64 bytes from 172.10.0.1: icmp_seq=2 ttl=64 time=0.254 ms
--- 172.10.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 31ms
rtt min/avg/max/mdev = 0.171/0.212/0.254/0.043 ms
# A. Server on vpp side
docker exec vpp-sut ip netns exec orange nstat -a
docker exec vpp-sut ip netns exec orange ncat -klv 172.10.0.1 12345&
docker exec vpp-sut ip netns exec orange ss -neptl
docker exec vpp-sut ip netns exec orange tcpdump -s0 -eni veth1 -w
/tmp/tcpdump/ext-in-tcp-failure-sut-veth1.pcap&
docker exec vpp-sut tcpdump -s0 -eni eth0 -w
/tmp/tcpdump/ext-in-tcp-failure-sut-eth1.pcap&
docker exec rvm-tester tcpdump -s0 -eni eth0.100 -w
/tmp/tcpdump/ext-in-tcp-failure-tester.pcap&
docker exec -ti rvm-tester ncat 172.10.0.1 12345
# After 'Ncat: Connection timed out'
docker exec vpp-sut ip netns exec orange nstat
#kernel
IpInReceives 4 0.0
IpInDelivers 4 0.0
TcpInSegs 4 0.0
TcpInErrs 4 0.0
TcpInCsumErrors 4 0.0
IpExtInOctets 240 0.0
IpExtInNoECTPkts 4 0.0
docker exec rvm-tester pkill tcpdump
docker exec vpp-sut pkill tcpdump
docker exec vpp-sut pkill ncat
# B. Client on vpp side
docker exec vpp-sut ip netns exec orange nstat -a
docker exec rvm-tester nstat -a
docker exec rvm-tester ncat -klv 172.10.0.253 9876&
docker exec rvm-tester ss -neptl
docker exec rvm-tester tcpdump -s0 -eni eth0.100 -w
/tmp/tcpdump/out-tcp-failure-tester.pcap&
docker exec vpp-sut ip netns exec orange tcpdump -s0 -eni veth1 -w
/tmp/tcpdump/out-tcp-failure-sut-veth1.pcap&
docker exec vpp-sut tcpdump -s0 -eni eth0 -w
/tmp/tcpdump/out-tcp-failure-sut-eth1.pcap&
docker exec -ti vpp-sut ip netns exec orange ncat 172.10.0.253 9876
# After 'Ncat: Connection timed out'
docker exec vpp-sut ip netns exec orange nstat
#kernel
IpInReceives 7 0.0
IpInDelivers 7 0.0
IpOutRequests 4 0.0
TcpActiveOpens 1 0.0
TcpInSegs 7 0.0
TcpOutSegs 1 0.0
TcpRetransSegs 3 0.0
TcpInErrs 7 0.0
TcpInCsumErrors 7 0.0
TcpExtTCPLostRetransmit 2 0.0
TcpExtTCPTimeouts 3 0.0
TcpExtTCPSynRetrans 3 0.0
TcpExtTcpTimeoutRehash 3 0.0
IpExtInOctets 420 0.0
IpExtOutOctets 240 0.0
IpExtInNoECTPkts 7 0.0
docker exec rvm-tester nstat
#kernel
IpInReceives 8 0.0
IpInDelivers 4 0.0
IpOutRequests 9 0.0
TcpInSegs 4 0.0
TcpOutSegs 9 0.0
TcpRetransSegs 8 0.0
TcpExtTCPTimeouts 5 0.0
TcpExtTCPSynRetrans 8 0.0
IpExtInOctets 1016 0.0
IpExtOutOctets 540 0.0
IpExtInNoECTPkts 8 0.0
(No errors on tester side)
docker exec rvm-tester pkill tcpdump
docker exec vpp-sut pkill tcpdump
docker exec vpp-sut pkill ncat
# Cleanup
docker stop vpp-sut
docker stop rvm-tester
out-tcp-failure-tester.pcap
Description: out-tcp-failure-tester.pcap
out-tcp-failure-sut-veth1.pcap
Description: out-tcp-failure-sut-veth1.pcap
out-tcp-failure-sut-eth1.pcap
Description: out-tcp-failure-sut-eth1.pcap
ext-in-tcp-failure-tester.pcap
Description: ext-in-tcp-failure-tester.pcap
ext-in-tcp-failure-sut-veth1.pcap
Description: ext-in-tcp-failure-sut-veth1.pcap
ext-in-tcp-failure-sut-eth1.pcap
Description: ext-in-tcp-failure-sut-eth1.pcap
---- SUT docker exec vpp-sut uname -a Linux 6a476928c79e 5.10.0-1053-oem #55-Ubuntu SMP Sun Dec 12 01:58:07 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux docker exec vpp-sut cat /etc/os-release NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/"; SUPPORT_URL="https://help.ubuntu.com/"; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"; VERSION_CODENAME=focal UBUNTU_CODENAME=focal docker exec vpp-sut vppctl show version verbose Version: v22.02-rc0~165-gb44d0defb Compiled by: root Compile host: c60037b8de21 Compile date: 2021-12-01T17:30:02 Compile location: /vpp Compiler: GCC 9.3.0 Current PID: 360 docker exec vpp-sut vppctl show hardware Name Idx Link Hardware host-eth0 1 up host-eth0 Link speed: unknown RX Queues: queue thread mode 0 main (0) interrupt Ethernet address 02:fe:b9:c3:f0:50 Linux PACKET socket interface TX block size:10485760 nr:1 TX frame size:10240 nr:1024 RX block size:10485760 nr:1 RX frame size:10240 nr:1024 next frame:19 available:1024 request:0 sending:0 wrong:0 total:1024 host-veth0 2 up host-veth0 Link speed: unknown RX Queues: queue thread mode 0 main (0) adaptive Ethernet address 02:fe:0e:e0:80:0c Linux PACKET socket interface TX block size:10485760 nr:1 TX frame size:10240 nr:1024 RX block size:10485760 nr:1 RX frame size:10240 nr:1024 next frame:30 available:1024 request:0 sending:0 wrong:0 total:1024 local0 0 down local0 Link speed: unknown local docker exec vpp-sut vppctl show interface address host-eth0 (up): host-eth0.100 (up): L2 xconnect host-veth0 host-veth0 (up): L2 xconnect host-eth0.100 local0 (dn): docker exec vpp-sut vppctl show interface host-eth0 1 up 9000/0/0/0 rx packets 164 rx bytes 29866 tx packets 19 tx bytes 1282 drops 134 ip4 129 ip6 5 host-eth0.100 3 up 0/0/0/0 rx packets 30 rx bytes 2164 tx packets 19 tx bytes 1282 host-veth0 2 up 9000/0/0/0 rx packets 19 rx bytes 1206 tx packets 30 tx bytes 2044 local0 0 down 0/0/0/0 docker exec vpp-sut vppctl show mode l3 local0 l3 host-eth0 l2 xconnect host-veth0 host-eth0.100 l2 xconnect host-eth0.100 host-veth0 docker exec vpp-sut vppctl show interface vtr host-eth0.100 host-eth0.100: VTR pop-1 docker exec vpp-sut vppctl show run Time 3995.6, 10 sec internal node vector rate 0.00 loops/sec 46222.47 vector rates in 4.7052e-2, out 1.2264e-2, drop 3.4789e-2, punt 0.0000e0 Name State Calls Vectors Suspends Clocks Vectors/Call acl-plugin-fa-cleaner-process event wait 0 0 1 1.50e4 0.00 af-packet-input interrupt wa 183 188 0 2.25e4 1.03 api-rx-from-ring any wait 0 0 203 6.44e4 0.00 avf-process event wait 0 0 1 2.32e4 0.00 bfd-process event wait 0 0 1 3.62e3 0.00 bond-process event wait 0 0 1 4.10e3 0.00 cnat-scanner-process event wait 0 0 1 6.57e3 0.00 dhcp-client-process any wait 0 0 4 9.07e3 0.00 dhcp6-client-cp-process any wait 0 0 1 3.08e3 0.00 dhcp6-pd-client-cp-process any wait 0 0 1 3.58e3 0.00 dhcp6-pd-reply-publisher-proce event wait 0 0 1 1.18e4 0.00 dhcp6-reply-publisher-process event wait 0 0 1 4.02e3 0.00 drop active 139 139 0 6.35e3 1.00 error-drop active 139 139 0 3.57e3 1.00 ethernet-input active 183 188 0 9.18e3 1.03 fib-walk any wait 0 0 1997 1.56e4 0.00 flow-report-process any wait 0 0 1 3.14e3 0.00 flowprobe-timer-process any wait 0 0 1 6.88e3 0.00 gbp-scanner event wait 0 0 1 6.69e3 0.00 host-eth0-output active 19 19 0 7.74e3 1.00 host-eth0-tx active 19 19 0 1.29e5 1.00 host-veth0-output active 30 30 0 5.36e3 1.00 host-veth0-tx active 30 30 0 8.84e4 1.00 igmp-timer-process event wait 0 0 1 9.39e3 0.00 ikev2-manager-process event wait 0 0 1 6.70e3 0.00 ioam-export-process any wait 0 0 1 3.29e3 0.00 ip-neighbor-event event wait 0 0 1 1.55e4 0.00 ip4-full-reassembly-expire-wal any wait 0 0 400 9.70e3 0.00 ip4-input active 133 133 0 1.17e4 1.00 ip4-neighbor-age-process event wait 0 0 1 2.83e3 0.00 ip4-not-enabled active 133 133 0 3.78e3 1.00 ip4-sv-reassembly-expire-walk any wait 0 0 400 6.28e3 0.00 ip6-full-reassembly-expire-wal any wait 0 0 400 5.42e3 0.00 ip6-input active 6 6 0 1.66e4 1.00 ip6-mld-process any wait 0 0 3994 7.68e3 0.00 ip6-neighbor-age-process event wait 0 0 1 3.51e3 0.00 ip6-not-enabled active 6 6 0 6.60e3 1.00 ip6-ra-process any wait 0 0 3994 7.62e3 0.00 ip6-rs-process any wait 0 0 1 3.18e3 0.00 ip6-sv-reassembly-expire-walk any wait 0 0 400 8.16e3 0.00 l2-arp-term-publisher event wait 0 0 1 5.19e3 0.00 l2-input active 44 49 0 8.54e3 1.11 l2-input-vtr active 30 30 0 3.93e3 1.00 l2-output active 44 49 0 4.98e3 1.11 l2fib-mac-age-scanner-process event wait 0 0 1 4.02e3 0.00 lldp-process event wait 0 0 1 6.21e4 0.00 memif-process event wait 0 0 1 7.84e3 0.00 nat44-ei-ha-process event wait 0 0 1 6.98e3 0.00 nsh-md2-ioam-export-process any wait 0 0 1 9.14e3 0.00 rd-cp-process any wait 0 0 1 1.23e4 0.00 send-dhcp6-client-message-proc any wait 0 0 1 7.46e3 0.00 send-dhcp6-pd-client-message-p any wait 0 0 1 6.86e3 0.00 startup-config-process done 1 0 1 1.29e4 0.00 statseg-collector-process time wait 0 0 400 4.08e6 0.00 udp-ping-process any wait 0 0 1 9.27e3 0.00 unix-cli-local:17 active 17 0 35 4.15e14 0.00 unix-cli-new-session any wait 0 0 31 6.58e7 0.00 unix-epoll-input polling 41640086 0 0 2.29e5 0.00 vhost-user-process any wait 0 0 1 3.32e3 0.00 vhost-user-send-interrupt-proc any wait 0 0 1 2.72e3 0.00 virtio-send-interrupt-process any wait 0 0 1 3.76e3 0.00 vpe-link-state-process event wait 0 0 7 3.79e3 0.00 vrrp-periodic-process event wait 0 0 1 1.52e4 0.00 vxlan-gpe-ioam-export-process any wait 0 0 1 5.31e3 0.00 wg-timer-manager event wait 0 0 1 2.62e7 0.00 docker exec vpp-sut vppctl show error Count Node Reason Severity 49 l2-output L2 output packets error 49 l2-input L2 input packets error 134 ethernet-input no error error ------ Tester docker exec rvm-tester uname -a Linux 8a85bbe50be0 5.10.0-1053-oem #55-Ubuntu SMP Sun Dec 12 01:58:07 UTC 2021 x86_64 GNU/Linux docker exec rvm-tester cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/"; SUPPORT_URL="https://www.debian.org/support"; BUG_REPORT_URL="https://bugs.debian.org/";
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20696): https://lists.fd.io/g/vpp-dev/message/20696 Mute This Topic: https://lists.fd.io/mt/88351189/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
