Anything new on this? I am only curios if we can count with this in 17.04 release.
Peter Mikus Engineer – Software Cisco Systems Limited From: Damjan Marion (damarion) Sent: Thursday, March 23, 2017 1:52 PM To: Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) <pmi...@cisco.com> Cc: vpp-dev <vpp-dev@lists.fd.io>; csit-...@lists.fd.io Subject: Re: VXLAN multithreading issue I think I know where it might be a problem. let me come back to you…. On 23 Mar 2017, at 13:08, Peter Mikus -X (pmikus - PANTHEON TECHNOLOGIES at Cisco) <pmi...@cisco.com<mailto:pmi...@cisco.com>> wrote: Hello vpp-dev, With latest VPP build packages I am observing the issue with VXLAN and LISPGPE tunnels during my testing (CSIT and internal Cisco lab - manual testing). I am getting the following errors: vpp# sh err Count Node Reason 10 ip4-udp-lookup no listener for dst port 10 ip4-icmp-error destination unreachable response sent 10 vxlan4-encap good packets encapsulated 10 l2-output L2 output packets 10 l2-learn L2 learn packets 10 l2-learn L2 learn hits 10 l2-input L2 input packets 10 l2-flood L2 flood packets Encapsulation of traffic into VXLAN is working properly but decapsulation is throwing "no listener for dst port" vpp# sh trace ------------------- Start of thread 0 vpp_main ------------------- No packets in trace buffer ------------------- Start of thread 1 vpp_wk_0 ------------------- Packet 1 00:04:49:119217: dpdk-input FortyGigabitEtherneta/0/0 rx queue 0 buffer 0x1d288a28: current data 14, length 94, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0 PKT MBUF: port 0, nb_segs 1, pkt_len 108 buf_len 2176, data_len 108, ol_flags 0x180, data_off 128, phys_addr 0x55264900 packet_type 0x291 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without extension headers RTE_PTYPE_L4_UDP (0x0200) UDP packet IP4: 00:00:71:51:52:d7 -> 3c:fd:fe:9d:1a:a0 UDP: 11.0.0.2 -> 11.0.0.1 tos 0x00, ttl 64, length 94, checksum 0x648d fragment id 0x0000 UDP: 4789 -> 4789 length 74, checksum 0x9c98 00:04:49:119239: ip4-input-no-checksum UDP: 11.0.0.2 -> 11.0.0.1 tos 0x00, ttl 64, length 94, checksum 0x648d fragment id 0x0000 UDP: 4789 -> 4789 length 74, checksum 0x9c98 00:04:49:119249: ip4-lookup fib 0 dpo-idx 5 flow hash: 0x00000000 UDP: 11.0.0.2 -> 11.0.0.1 tos 0x00, ttl 64, length 94, checksum 0x648d fragment id 0x0000 UDP: 4789 -> 4789 length 74, checksum 0x9c98 00:04:49:119253: ip4-local UDP: 11.0.0.2 -> 11.0.0.1 tos 0x00, ttl 64, length 94, checksum 0x648d fragment id 0x0000 UDP: 4789 -> 4789 length 74, checksum 0x9c98 00:04:49:119256: ip4-udp-lookup UDP: src-port 4789 dst-port 4789 (no listener) 00:04:49:119261: ip4-icmp-error UDP: 11.0.0.2 -> 11.0.0.1 tos 0x00, ttl 64, length 94, checksum 0x648d fragment id 0x0000 UDP: 4789 -> 4789 length 74, checksum 0x9c98 00:04:49:119265: ip4-lookup fib 0 dpo-idx 2 flow hash: 0x00000000 ICMP: 11.0.0.1 -> 11.0.0.2 tos 0x00, ttl 255, length 122, checksum 0xa580 fragment id 0x0000 ICMP destination_unreachable port_unreachable checksum 0x135b 00:04:49:119265: ip4-rewrite tx_sw_if_index 1 dpo-idx 2 : ipv4 via 11.0.0.2 FortyGigabitEtherneta/0/0: IP4: 3c:fd:fe:9d:1a:a0 -> 00:00:71:51:52:d7 flow hash: 0x00000000 IP4: 3c:fd:fe:9d:1a:a0 -> 00:00:71:51:52:d7 ICMP: 11.0.0.1 -> 11.0.0.2 tos 0x00, ttl 254, length 122, checksum 0xa680 fragment id 0x0000 ICMP destination_unreachable port_unreachable checksum 0x135b 00:04:49:119266: FortyGigabitEtherneta/0/0-output FortyGigabitEtherneta/0/0 IP4: 3c:fd:fe:9d:1a:a0 -> 00:00:71:51:52:d7 ICMP: 11.0.0.1 -> 11.0.0.2 tos 0x00, ttl 254, length 122, checksum 0xa680 fragment id 0x0000 ICMP destination_unreachable port_unreachable checksum 0x135b 00:04:49:119268: FortyGigabitEtherneta/0/0-tx FortyGigabitEtherneta/0/0 tx queue 1 buffer 0x1d288a28: current data -28, length 136, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0 IP4: 3c:fd:fe:9d:1a:a0 -> 00:00:71:51:52:d7 ICMP: 11.0.0.1 -> 11.0.0.2 tos 0x00, ttl 254, length 122, checksum 0xa680 fragment id 0x0000 ICMP destination_unreachable port_unreachable checksum 0x135b ------------------- Start of thread 2 vpp_wk_1 ------------------- Packet 1 00:04:49:119215: dpdk-input FortyGigabitEtherneta/0/1 rx queue 0 buffer 0x1d292712: current data 0, length 60, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0 PKT MBUF: port 1, nb_segs 1, pkt_len 60 buf_len 2176, data_len 60, ol_flags 0x180, data_off 128, phys_addr 0x554d8380 packet_type 0x691 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without extension headers RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet IP4: 00:00:22:22:00:00 -> 00:00:11:11:00:00 unknown 61: 32.0.0.0 -> 16.0.0.0 tos 0x00, ttl 64, length 46, checksum 0x4a94 fragment id 0x0000 00:04:49:119239: ethernet-input IP4: 00:00:22:22:00:00 -> 00:00:11:11:00:00 00:04:49:119247: l2-input l2-input: sw_if_index 2 dst 00:00:11:11:00:00 src 00:00:22:22:00:00 00:04:49:119252: l2-learn l2-learn: sw_if_index 2 dst 00:00:11:11:00:00 src 00:00:22:22:00:00 bd_index 0 00:04:49:119255: l2-fwd l2-fwd: sw_if_index 2 dst 00:00:11:11:00:00 src 00:00:22:22:00:00 bd_index 0 00:04:49:119257: l2-flood l2-flood: sw_if_index 2 dst 00:00:11:11:00:00 src 00:00:22:22:00:00 bd_index 0 00:04:49:119258: l2-output l2-output: sw_if_index 3 dst 00:00:11:11:00:00 src 00:00:22:22:00:00 data 08 00 45 00 00 2e 00 00 00 00 40 3d 00:04:49:119261: vxlan4-encap VXLAN encap to vxlan_tunnel0 vni 101 00:04:49:119264: ip4-load-balance fib 3 dpo-idx 10 flow hash: 0x00000000 UDP: 11.0.0.1 -> 11.0.0.2 tos 0x00, ttl 254, length 96, checksum 0xa68a fragment id 0x0000 UDP: 57494 -> 4789 length 76, checksum 0x0000 00:04:49:119267: ip4-rewrite tx_sw_if_index 1 dpo-idx 2 : ipv4 via 11.0.0.2 FortyGigabitEtherneta/0/0: IP4: 3c:fd:fe:9d:1a:a0 -> 00:00:71:51:52:d7 flow hash: 0x00000000 IP4: 3c:fd:fe:9d:1a:a0 -> 00:00:71:51:52:d7 UDP: 11.0.0.1 -> 11.0.0.2 tos 0x00, ttl 253, length 96, checksum 0xa78a fragment id 0x0000 UDP: 57494 -> 4789 length 76, checksum 0x0000 00:04:49:119268: FortyGigabitEtherneta/0/0-output FortyGigabitEtherneta/0/0 IP4: 3c:fd:fe:9d:1a:a0 -> 00:00:71:51:52:d7 UDP: 11.0.0.1 -> 11.0.0.2 tos 0x00, ttl 253, length 96, checksum 0xa78a fragment id 0x0000 UDP: 57494 -> 4789 length 76, checksum 0x0000 00:04:49:119268: FortyGigabitEtherneta/0/0-tx FortyGigabitEtherneta/0/0 tx queue 2 buffer 0x1d292712: current data -50, length 110, free-list 0, clone-count 0, totlen-nifb 0, trace 0x0 IP4: 3c:fd:fe:9d:1a:a0 -> 00:00:71:51:52:d7 UDP: 11.0.0.1 -> 11.0.0.2 tos 0x00, ttl 253, length 96, checksum 0xa78a fragment id 0x0000 UDP: 57494 -> 4789 length 76, checksum 0x0000 # Startup - Config: unix { nodaemon cli-listen localhost:5002 startup-config /home/cisco/vppcfg-init/vpp-2p1nic40ge-ethip4vxlan-l2bdbasemaclrn-asymmetric.conf } cpu { main-core 7 corelist-workers 1,33 } api-trace { on } dpdk { dev default { num-rx-queues 1 } dev 0000:0a:00.0 { workers 0 } dev 0000:0a:00.1 { workers 1 } socket-mem 8192,8192 no-multi-seg num-mbufs 512000 } # Init - Config: set interface state FortyGigabitEtherneta/0/0 up set interface state FortyGigabitEtherneta/0/1 up set interface ip address FortyGigabitEtherneta/0/0 11.0.0.1/24 set ip arp FortyGigabitEtherneta/0/0 11.0.0.2 0000.7151.52d7 create vxlan tunnel src 11.0.0.1 dst 11.0.0.2 vni 101 set interface l2 bridge vxlan_tunnel0 0 set interface l2 bridge FortyGigabitEtherneta/0/1 0 set bridge-domain forward 0 set bridge-domain flood 0 set bridge-domain learn 0 set bridge-domain uu-flood 0 # Gerrit bisecting with testing: #5666 https://gerrit.fd.io/r/#/c/5666/ - Python API: Change from cPython to CFFI. (Last version with VXLAN/LISPGPE working) #5746 https://gerrit.fd.io/r/#/c/5746/ - vlib: make runtime_data thread-local (Unable to test - crashing) #5700 https://gerrit.fd.io/r/#/c/5700/ - vhost: wrong value return for VHOST_USER_VRING_GET_BASE (Unable to test - crashing; unlilkely the root cause) #5765 https://gerrit.fd.io/r/#/c/5765/ - API:replaced all REPLY_MACRO's with api_helper_macros.h (Unable to test - crashing) #5770 https://gerrit.fd.io/r/#/c/5770/ - ping: Fix coverity error 163907 (Unable to test - crashing; unlilkely the root cause) #5778 https://gerrit.fd.io/r/#/c/5778/ - vlib: additional runtime_data checks (VXLAN/LISPGPE not working) After investigation I found that configuring VPP without cpu section, it is working (looks like thread mapping issue). This is why the CSIT - VIRL was passing the tests. unix { nodaemon cli-listen localhost:5002 startup-config /home/cisco/vppcfg-init/vpp-2p1nic40ge-ethip4vxlan-l2bdbasemaclrn-asymmetric.conf } api-trace { on } dpdk { dev default { num-rx-queues 1 } dev 0000:0a:00.0 dev 0000:0a:00.1 socket-mem 8192,8192 no-multi-seg num-mbufs 512000 } Can you please take a look and help? Thank you. Peter Mikus Engineer – Software Cisco Systems Limited
_______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
