Hi Neale, The strongswan (sender) has configured tunnel mode SA. The below is the config taken from SS ipsec.conf file that indicates tunnel Mode: - *"type=tunnel"*
If the SS has configured tunnel mode and is sending proto = ESP (not proto =IP) then should the tunnel src/tunnel dst of the pkt be exactly the same as the addresses in the show ipip cmd (in my setup these two addresses were opposite)? On Mon, Mar 8, 2021 at 6:49 PM Neale Ranns <ne...@graphiant.com> wrote: > > > Hi Vijay, > > > > I always add the list to correspondence otherwise the list only sees the > question not the answer. > > > > Your tunnel looks fine. Its source address matches a VPP interface, and > packets are destined to it. However, packets don’t classify to the tunnel, > because they arrive proto=ESP not proto=IP, which implies the sender has > configured transport mode SAs. > > > > /neale > > > > > > > > *From: *Vijay Kumar <vjkumar2...@gmail.com> > *Date: *Monday, 8 March 2021 at 13:32 > *To: *Neale Ranns <ne...@graphiant.com> > *Subject: *Re: [vpp-dev] Traffic is not put on IPSec tunnel intf ipip0 > > Hi Neale, > > > > I didnt want to flood others' inbox hence unicasting the reply. This is my > topology > > > > SS (20.20.99.215) < ==============================> VPP (20.20.99.99) > > > > As you mentioned in your reply, my tunnel src (as seen in show ipip > tunnel o/p) is pointing to an interface address that is hosted in the VPP > device which is 20.20.99.99 in this case. But the ping is coming from > 20.20.99.215 which is that of that SS. > > > > *Is my understanding wrong?* Did u mean the ipip tunnel src must be > 20.20.99.215 (SS) > > > > > > *vpp#show trace* > 00:04:10:067403: dpdk-input > VirtualFuncEthernet0/6/0 rx queue 0 > buffer 0x4c3000: current data 0, length 170, buffer-pool 0, ref-count 1, > totlen-nifb 0, trace handle 0x1000000 > ext-hdr-valid > l4-cksum-computed l4-cksum-correct > PKT MBUF: port 0, nb_segs 1, pkt_len 170 > buf_len 2176, data_len 170, ol_flags 0x180, data_off 128, phys_addr > 0x50ac0080 > packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > Packet Offload Flags > PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid > PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid > Packet Types > RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet > RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without > extension headers > RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet > IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:9a:2c:38 802.1q vlan 1556 > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067423: ethernet-input > frame: flags 0x3, hw-if-index 3, sw-if-index 3 > IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:9a:2c:38 802.1q vlan 1556 > 00:04:10:067429: ip4-input > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067435: ip4-lookup > fib 1 dpo-idx 24 flow hash: 0x00000000 > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067439: ip4-local > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067440: ip4-punt > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067443: error-punt > rx:VirtualFuncEthernet0/6/0.1556 > 00:04:10:067448: punt > ip4-local: unknown ip protocol > > > vpp# show ipip tun > tunnel tunnel-hash > vpp# show ipip tunnel > *[0] instance 0 src 20.20.99.99 dst 20.20.99.215 table-ID 0 sw-if-idx 18 > flags [none] dscp CS0* > vpp# > > > > On Mon, Mar 8, 2021 at 2:55 PM Neale Ranns <ne...@graphiant.com> wrote: > > Hi Vijay, > > > > When you create an ip-ip tunnel the source address is the one that is > local to the device on which you are configuring the tunnel. The > destination address is that of the peer. Given packets arrive at your > device with address 20.20.99.99, you tunnel addresses are backwards. > > > > /neale > > > > > > *From: *Vijay Kumar <vjkumar2...@gmail.com> > *Date: *Monday, 8 March 2021 at 10:20 > *To: *Neale Ranns <ne...@graphiant.com> > *Subject: *Re: [vpp-dev] Traffic is not put on IPSec tunnel intf ipip0 > > Hi Neale, > > > > The ipip tunnel src points to the VTH IP hosted in VPP while ipip tunnel > dst points to an interface hosted in the peer VM (Strongswan). The ESP pkt > in the trace has source ip that points to the IP of Strongswan. > > > > I have pasted the output of trace and show ipip. The IPs in the pkt trace > output is opposite (reversed) to the IPs in the show ipip tunnel cmd. This > could be because my traffic direction right (SS is starting traffic)? Or is > this a problem > > > > *NOTE: -* > > I created ipip interface and installed SA using the same logic as > in ikev2_add_tunnel_from_main() > > > > *vpp#show trace* > 00:04:10:067403: dpdk-input > VirtualFuncEthernet0/6/0 rx queue 0 > buffer 0x4c3000: current data 0, length 170, buffer-pool 0, ref-count 1, > totlen-nifb 0, trace handle 0x1000000 > ext-hdr-valid > l4-cksum-computed l4-cksum-correct > PKT MBUF: port 0, nb_segs 1, pkt_len 170 > buf_len 2176, data_len 170, ol_flags 0x180, data_off 128, phys_addr > 0x50ac0080 > packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > Packet Offload Flags > PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid > PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid > Packet Types > RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet > RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without > extension headers > RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet > IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:9a:2c:38 802.1q vlan 1556 > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067423: ethernet-input > frame: flags 0x3, hw-if-index 3, sw-if-index 3 > IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:9a:2c:38 802.1q vlan 1556 > 00:04:10:067429: ip4-input > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067435: ip4-lookup > fib 1 dpo-idx 24 flow hash: 0x00000000 > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067439: ip4-local > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067440: ip4-punt > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0x5757 dscp CS0 ecn NON_ECN > fragment id 0xf37a, flags DONT_FRAGMENT > 00:04:10:067443: error-punt > rx:VirtualFuncEthernet0/6/0.1556 > 00:04:10:067448: punt > ip4-local: unknown ip protocol > > > vpp# show ipip tun > tunnel tunnel-hash > vpp# show ipip tunnel > *[0] instance 0 src 20.20.99.99 dst 20.20.99.215 table-ID 0 sw-if-idx 18 > flags [none] dscp CS0* > vpp# > > > > On Mon, Mar 8, 2021 at 2:31 PM Neale Ranns <ne...@graphiant.com> wrote: > > Hi Vijay, > > > > VPP drops because the packets don’t classify to your tunnel, not because > ESP is not registered. Compare the addresses in the output from ‘sh ipip > tun’ with the packet in the trace. > > > > /neale > > > > > > *From: *Vijay Kumar <vjkumar2...@gmail.com> > *Date: *Monday, 8 March 2021 at 09:52 > *To: *Neale Ranns <ne...@graphiant.com> > *Cc: *vpp-dev <vpp-dev@lists.fd.io> > *Subject: *Re: [vpp-dev] Traffic is not put on IPSec tunnel intf ipip0 > > Hi Neale, > > > > Yes, you are right. > > > > This is the output of show ip local cmd. The IPSEC_ESP is not registered. > Hence the drop. > > NOTE: - > > =========== > > I have a working POC setup that has IKEv2 plugin enabled. In this POC > setup, I can find IPSEC_ESP in the show ip local output. > > On my setup, I disabled the plugin ikev2. I only have a vnet ipsec graph > node (but no ikev2). The ike stack is running on a separate VM which is > that of Azure ike stack. > > > > > > *vpp# show ip local* > Protocols handled by ip4_local > ICMP: ip4-icmp-input > IGMP: igmp-input > IP_IN_IP: ipip4-input > TCP: an_ppe_router_input > UDP: ip4-udp-lookup > IPV6: ipip4-input > VRRP: vrrp4-input > unknown 200: an_ppe_wfectrl > unknown 201: an_ppe_vppctrl_input > vpp# > vpp# > > > > On Mon, Mar 8, 2021 at 2:08 PM Neale Ranns <ne...@graphiant.com> wrote: > > Hi Vijay, > > > > ‘unknown IP protocol’ means there is no registered handler for ESP. From > the info you gave I am assuming you are using ip-ip tunnel protection with > the SAs in transport mode. In that case the incoming packet should classify > to the ip-ip tunnel. The fact that it doesn’t, and the fact that the per > through the tunnel is unreachable, suggests that the src,dst addresses on > this tunnel are incorrect. The VPP IKE plugin would have set these > addresses correctly for you based on the IKE session end point. > > > > /neale > > > > > > *From: *vpp-dev@lists.fd.io <vpp-dev@lists.fd.io> on behalf of Vijay > Kumar via lists.fd.io <vjkumar2003=gmail....@lists.fd.io> > *Date: *Monday, 8 March 2021 at 08:44 > *To: *vpp-dev <vpp-dev@lists.fd.io> > *Subject: *Re: [vpp-dev] Traffic is not put on IPSec tunnel intf ipip0 > > Hi, > > > > Below is the trace output for "unknown ip protocol" related drops for the > ESP pkts coming from the strongswan to VPP. > > Let me know If I need to add some ip4-punt related config. > > > > show trace o/p > > =============== > > 00:08:29:451365: dpdk-input > VirtualFuncEthernet0/7/0 rx queue 0 > buffer 0x4c2d69: current data 0, length 170, buffer-pool 0, ref-count 1, > totlen-nifb 0, trace handle 0x1000001 > ext-hdr-valid > l4-cksum-computed l4-cksum-correct > PKT MBUF: port 0, nb_segs 1, pkt_len 170 > buf_len 2176, data_len 170, ol_flags 0x180, data_off 128, phys_addr > 0x4ecb5ac0 > packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 > rss 0x0 fdir.hi 0x0 fdir.lo 0x0 > Packet Offload Flags > PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid > PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid > Packet Types > RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet > RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without > extension headers > RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet > IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:c2:b4:f4 802.1q vlan 1556 > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0xbe54 dscp CS0 ecn NON_ECN > fragment id 0x8c7d, flags DONT_FRAGMENT > 00:08:29:451372: ethernet-input > frame: flags 0x3, hw-if-index 3, sw-if-index 3 > IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:c2:b4:f4 802.1q vlan 1556 > 00:08:29:451377: ip4-input > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0xbe54 dscp CS0 ecn NON_ECN > fragment id 0x8c7d, flags DONT_FRAGMENT > 00:08:29:451380: ip4-lookup > fib 1 dpo-idx 27 flow hash: 0x00000000 > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0xbe54 dscp CS0 ecn NON_ECN > fragment id 0x8c7d, flags DONT_FRAGMENT > 00:08:29:451382: ip4-local > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0xbe54 dscp CS0 ecn NON_ECN > fragment id 0x8c7d, flags DONT_FRAGMENT > 00:08:29:451383: ip4-punt > IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 > tos 0x00, ttl 64, length 152, checksum 0xbe54 dscp CS0 ecn NON_ECN > fragment id 0x8c7d, flags DONT_FRAGMENT > 00:08:29:451384: error-punt > rx:VirtualFuncEthernet0/7/0.1556 > 00:08:29:451386: punt > * ip4-local: unknown ip protocol* > > > > On Mon, Mar 8, 2021 at 2:00 AM Vijay Kumar via lists.fd.io <vjkumar2003= > gmail....@lists.fd.io> wrote: > > Hi, > > > > Strongswan <======================> [VPP IPSec] + [Azure IKE stack] > > > > I have a use case like above. Strongswan is the peer while VPP is the > responder. On the right hand side, I am not showing VPP ikev2 plugin > because we have disabled the plugin but rather using the Microsoft Azure > ike stack. In our case the ike stack and vpp run on different pods. We have > written a new plugin for IKE pkt forwarding to the pod where IKE stack is > running. The new plugin registers for udp port 500 and 4500 and thus fwds > all the received IKE traffic to IKE stack (to be precise, the new VPP > plugin forwards IKE packets to the application process via memif. The > application process in turn uses REST API to forward the IKE pkts to the > pods that runs the IKE stack). Once SA is established, the IKE stack would > install the SA on to the VPP IPSec plugin using VPP APIs (we would use > the vl_api_rpc_call_main_thread function call to execute the REST API SA > information). > > > > The *IPSec SA is established fine* as displayed below but ping to the > host on the strongswan side is failing as "*blackholed packets*" > > > > CLIs logs > > ================ > > vpp# show ipsec sa > [0] sa 256349522 (0xf479552) spi 3475331116 (0xcf25582c) protocol:esp > flags:[anti-replay ] > [1] sa 256349523 (0xf479553) spi 305419899 (0x1234567b) protocol:esp > flags:[anti-replay inbound ] > vpp# > > NOTE > > ============= > > 20.200.75.1 is the host on the strongswan side, > while VirtualFuncEthernet0/7/0.1556 is the interface that is connected to > the Strongswan (this interface is pingable with the SS interface) > > > > vpp# show interface > Name Idx State MTU (L3/IP4/IP6/MPLS) Counter > Count > VirtualFuncEthernet0/7/0 3 up 1500/0/0/0 rx > packets 12 > rx > bytes 2432 > tx > packets 12 > tx > bytes 2408 > drops > 8 > VirtualFuncEthernet0/7/0.1556 14 up 1500/0/0/0 rx > packets 30 > rx > bytes 647112 > tx > packets 12 > tx > bytes 2408 > drops > 8 > ip4 > 8 > VirtualFuncEthernet0/9/0 4 down 9000/0/0/0 > ipip0 18 down 9000/0/0/0 > local0 0 down 0/0/0/0 > > > > > > vpp# set interface ip address ipip0 20.200.76.2/32 > > vpp# set interface state ipip0 up > vpp# ip route add 20.200.75.1/32 via ipip0 > vpp# set interface unnumbered ipip0 use VirtualFuncEthernet0/7/0.1556 > > > > vpp# show interface > Name Idx State MTU (L3/IP4/IP6/MPLS) > Counter Count > VirtualFuncEthernet0/7/0 3 up 1500/0/0/0 rx > packets 14 > rx > bytes 2618 > tx > packets 14 > tx > bytes 2594 > drops > 9 > VirtualFuncEthernet0/7/0.1556 14 up 1500/0/0/0 rx > packets 34 > rx > bytes 711766 > tx > packets 14 > tx > bytes 2594 > drops > 9 > ip4 > 9 > VirtualFuncEthernet0/9/0 4 down 9000/0/0/0 > ipip0 18 up 9000/0/0/0 > local0 0 down 0/0/0/0 > > > > > > *ISSUE* > > *=========* > > All the ping traffic from SS are coming to VPP as ESP packets but the VPP > is dropping them as "*unknown ip protocol*". Also, for all the ping > packets triggered from VPP side are not being put onto the ipip0 IPSec > tunnel but they are dropped as "*blackholed packets*" > > > > *NOTE: -* > > *==============* > > On the VPP, after adding a route to the destination 20.200.75.1, the show > ip FIB output shows dpo-drop rather than send via > "VirtualFuncEthernet0/7/0.1556" as shown below. > > > > > > *vpp# show if fib* > > > > > > > *20.200.75.1/32 <http://20.200.75.1/32> unicast-ip4-chain [@0]: > dpo-load-balance: [proto:ip4 index:81 buckets:1 uRPF:109 to:[0:0]] [0] > [@6]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:16 > 450000000000000040048b9814146363141463d7 stacked-on entry:78: > [@1]: dpo-drop ip4* > > > > > > vpp# show node counters > Count Node Reason > * 10 null-node blackholed packets* > 81 memif-input not ip packet > 2731 an_ppe_wfectrl wfectrl packets received > 2731 an_ppe_wfectrl wfectrl replies sent > 2181 an_ppe_wfectrl session stat request > received > 1 an_ppe_wfectrl service construct config > request received > 1 an_ppe_wfectrl service construct config > request success > 1 an_ppe_wfectrl service config request > received > 1 an_ppe_wfectrl service config request > success > 453 an_ppe_wfectrl dpi stats request > received > 453 an_ppe_wfectrl dpi stats request success > 19 an_ppe_wfectrl nat stats request > received > 19 an_ppe_wfectrl nat stats request success > 58 arp-reply ARP replies sent > 81 ip4-udp-lookup No error > * 14 ip4-local unknown ip protocol* > vpp# > vpp# > > > > > > *NOTE* > > *=============* > > When I did a POC using VPP ikev2 plugin and VPP IPSec, I did not face this > issue. The ping traffic from SS would be successfully decrypted at VPP and > also the reply sent back to SS. In the POC, the route entry of the > destination was fine. It was not dpo-drop. > > > > > > Please suggest if there is any way to debug the traffic drops as "Unknown > ip protocol" for incoming ESP traffic and traffic drops as "blackholed > packets" for outgoing traffic > > > > Any help is highly appreciated. > > > > > > Regards, > > Vijay Kumar N > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#18886): https://lists.fd.io/g/vpp-dev/message/18886 Mute This Topic: https://lists.fd.io/mt/81158278/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
