Hi Vijay, Are your protecting SAs in transport mode?. see: https://wiki.fd.io/view/VPP/IPSec#Protection_Model
/neale From: Vijay Kumar <vjkumar2...@gmail.com> Date: Friday, 26 March 2021 at 02:13 To: Neale Ranns <ne...@graphiant.com> Cc: vpp-dev <vpp-dev@lists.fd.io> Subject: Re: [vpp-dev] GRE-over-IPSec fails Hi Neale, Is this issue due to adjacency being wrong. Is this a day one issue or is this issue due to the way I added routes in my test case. As far as I know the way I configured GRE peer and routing seems to be correct. Did you see this issue anytime on your setup? Regards. On Thu, 25 Mar 2021, 19:41 Vijay Kumar, <vjkumar2...@gmail.com<mailto:vjkumar2...@gmail.com>> wrote: Hi Neale, I was able to make a good progress for the GRE-over-IPSec use case. But stumbled at the last step. I have explained the problem faced (an extra IP header added by VPP, overall 4 IP headers in the pkt) and other details about logs and configuration below. Also attached the decrypted pcap (frame 4 shows the problem) Kindly let me know if you can catch the problem at the VPP side. Topology ============ Strongswan <====================================> VPP loopback 7.7.7.7 loopback 8.8.8.8 GRE 44.44.44.44 GRE src 42.42.42.42 (tunnel remote 44.44.44.44) IPSEC 20.20.99.215 IPSEC 20.20.99.99 create gre tunnel src 42.42.42.42 instance 1 multipoint set interface state gre1 up set interface ip addr gre1 2.2.2.2/32<http://2.2.2.2/32> create teib gre1 peer 2.2.2.1 nh 44.44.44.44 ip route add 7.7.7.7/32<http://7.7.7.7/32> via 2.2.2.1 gre1 (added route to the overlay 7.7.7.7 of SS) /* added below commands after IPSec is UP */ set interface state ipip0 up ip route add 44.44.44.44/32<http://44.44.44.44/32> via ipip0 set interface unnumbered ipip0 use VirtualFuncEthernet0/7/0.1556 Problem faced ============== --- SS sends ICMP over GRE over IPSEC. The VPP also replies with ICMP-o-GRE-o-IPSEC. But the SS was dropping the packets. Debugging the SS logs did not show any decryption failure. I decrypted the packets captured at SS. On applying the encrytion and auth keys in the wireshark, it was showing an extra IP header added to the reply packet making it a total of 4 IP headers which is wrong --- I observed that VPP had added GRE tunnel IP header on top of the inner most packet. Again on top of the GRE IP header, another IP was added by VPP with SRC and DST being same as that of IPSec tunnel header (20.20.99.99-to-20.20.99.215). Now on top of this extra IP header the ESP encapsulation was added with outermost IP header IPs of that of the IPSec tunnel (20.20.99.99-to-20.20.99.215) --- Why is VPP adding an extra header on top of GRE IP header. Is this because of the route adjacencies that has gone bad causing an extra IP header to be added? --- Below I have pasted the FIB entry for the destination 7.7.7.7 that are stacked on FIB entry 73 and 75 --- Attached IPSec decoded pcap, please check frame 4 that is the ICMP reply from VPP to SS --- Is there anything wrong in the way I have added routes on the VPP side for the overlay 7.7.7.7 and GRE destination 44.44.44.44 Ping on SS ======== root@SS# ping 8.8.8.8 -I 7.7.7.7 -c 3 PING 8.8.8.8 (8.8.8.8) from 7.7.7.7 : 56(84) bytes of data. --- 8.8.8.8 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2041ms TCPDUMP taken on SS ======================= 12:46:48.439122 IP vmsrvrlnx-strongswan-215 > dns.google: ICMP echo request, id 43, seq 1, length 64 12:46:48.439174 IP vmsrvrlnx-strongswan-215 > 20.20.99.99<http://20.20.99.99>: ESP(spi=0x1234567b,seq=0x1), length 148 12:46:48.439178 ethertype IPv4, IP vmsrvrlnx-strongswan-215 > 20.20.99.99<http://20.20.99.99>: ESP(spi=0x1234567b,seq=0x1), length 148 12:46:48.439487 ethertype IPv4, IP 20.20.99.99 > vmsrvrlnx-strongswan-215: ESP(spi=0xce90b744,seq=0x1), length 180 12:46:48.439487 IP 20.20.99.99 > vmsrvrlnx-strongswan-215: ESP(spi=0xce90b744,seq=0x1), length 180 12:46:49.455775 IP vmsrvrlnx-strongswan-215 > dns.google: ICMP echo request, id 43, seq 2, length 64 12:46:49.455826 IP vmsrvrlnx-strongswan-215 > 20.20.99.99<http://20.20.99.99>: ESP(spi=0x1234567b,seq=0x2), length 148 12:46:49.455831 ethertype IPv4, IP vmsrvrlnx-strongswan-215 > 20.20.99.99<http://20.20.99.99>: ESP(spi=0x1234567b,seq=0x2), length 148 12:46:49.455958 ethertype IPv4, IP 20.20.99.99 > vmsrvrlnx-strongswan-215: ESP(spi=0xce90b744,seq=0x2), length 180 12:46:49.455958 IP 20.20.99.99 > vmsrvrlnx-strongswan-215: ESP(spi=0xce90b744,seq=0x2), length 180 12:46:50.479872 IP vmsrvrlnx-strongswan-215 > dns.google: ICMP echo request, id 43, seq 3, length 64 12:46:50.479933 IP vmsrvrlnx-strongswan-215 > 20.20.99.99<http://20.20.99.99>: ESP(spi=0x1234567b,seq=0x3), length 148 12:46:50.479939 ethertype IPv4, IP vmsrvrlnx-strongswan-215 > 20.20.99.99<http://20.20.99.99>: ESP(spi=0x1234567b,seq=0x3), length 148 12:46:50.480080 ethertype IPv4, IP 20.20.99.99 > vmsrvrlnx-strongswan-215: ESP(spi=0xce90b744,seq=0x3), length 180 12:46:50.480080 IP 20.20.99.99 > vmsrvrlnx-strongswan-215: ESP(spi=0xce90b744,seq=0x3), length 180 VPP LOGS ================================= vpp# show gre tunnel [0] instance 1 src 42.42.42.42 dst 0.0.0.0 fib-idx 0 sw-if-idx 20 payload L3 multi-point vpp# show teib [0] gre1:2.2.2.1 via [0]:44.44.44.44/32<http://44.44.44.44/32> vpp# show ipip tunnel [0] instance 0 src 20.20.99.99 dst 20.20.99.215 table-ID 0 sw-if-idx 21 flags [none] dscp CS0 FIB ouput of loopback IPs and GRE tunnel IPs shown in topology ============================================= 7.7.7.7/32<http://7.7.7.7/32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:75 buckets:1 uRPF:105 to:[0:0]] [0] [@6]: ipv4 via 2.2.2.1 gre1: mtu:9000 next:15 4500000000000000fe2f10232a2a2a2a2c2c2c2c00000800 stacked-on entry:73: [@3]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:17 450000000000000040048b9814146363141463d7 stacked-on entry:75: [@2]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556: mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800 8.8.8.8/32<http://8.8.8.8/32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:72 buckets:1 uRPF:103 to:[0:0]] [0] [@2]: dpo-receive: 8.8.8.8 on loop3 42.42.42.42/32<http://42.42.42.42/32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:71 buckets:1 uRPF:102 to:[0:0]] [0] [@2]: dpo-receive: 42.42.42.42 on loop2 44.44.44.44/32<http://44.44.44.44/32> unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:74 buckets:1 uRPF:107 to:[0:0]] [0] [@6]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:17 450000000000000040048b9814146363141463d7 stacked-on entry:75: [@2]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556: mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800 vpp# vpp# show fib entry 73 73@44.44.44.44/32<http://73@44.44.44.44/32> fib:0 index:73 locks:4 CLI refs:1 entry-flags:attached, src-flags:added,contributing,active, path-list:[81] locks:2 flags:shared, uPRF-list:107 len:1 itfs:[21, ] path:[117] pl-index:81 ip4 weight=1 pref=0 attached-nexthop: oper-flags:resolved, cfg-flags:attached, 44.44.44.44 ipip0 (p2p) [@0]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:17 450000000000000040048b9814146363141463d7 stacked-on entry:75: [@2]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556: mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800 recursive-resolution refs:1 src-flags:added, cover:-1 forwarding: unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:74 buckets:1 uRPF:107 to:[0:0]] [0] [@6]: ipv4 via 0.0.0.0 ipip0: mtu:9000 next:17 450000000000000040048b9814146363141463d7 stacked-on entry:75: [@2]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556: mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800 Delegates: track: sibling:147 Children:{adj:17} Children:{fib-entry-track:16} vpp# vpp# vpp# vpp# show fib entry 75 75@20.20.99.215/32<http://75@20.20.99.215/32> fib:0 index:75 locks:4 adjacency refs:1 entry-flags:attached, src-flags:added,contributing,active, cover:65 path-list:[80] locks:2 uPRF-list:106 len:1 itfs:[17, ] path:[116] pl-index:80 ip4 weight=1 pref=0 attached-nexthop: oper-flags:resolved, 20.20.99.215 VirtualFuncEthernet0/7/0.1556 [@0]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556: mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800 Extensions: path:116 adj-flags:[refines-cover] recursive-resolution refs:1 src-flags:added, cover:-1 forwarding: unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:76 buckets:1 uRPF:106 to:[14:3252]] [0] [@5]: ipv4 via 20.20.99.215 VirtualFuncEthernet0/7/0.1556: mtu:9000 next:16 fa163e4b6b42fa163ec2b4f4810006140800 Delegates: track: sibling:157 Children:{adj:19}{ipsec-sa:0} Children:{fib-entry-track:17} vpp# vpp# Regards, Vijay On Tue, Mar 23, 2021 at 5:44 PM Neale Ranns <ne...@graphiant.com<mailto:ne...@graphiant.com>> wrote: Hi Vijay, I don’t think you can have both a GRE and IPIP tunnel to the same peer, sourced from the same local address. These two tunnels will create two identical ‘keys’ in the lookup table (src:X,dst:Y,proto:ESP). my guess is your packet matches against the IPIP tunnel, not the GRE and the key material is different, hence the integ failure. You can see in the trace that it used SA:1, check ‘sh ipsec sa 1’ and ‘sh ipsec protect’ to see which tunnel that is associated with. You can also see the lookup table with ‘sh ipsec protect-hash’. If you remove (or admin down) the ipip tunnel, does it work? /neale From: Vijay Kumar <vjkumar2...@gmail.com<mailto:vjkumar2...@gmail.com>> Date: Tuesday, 23 March 2021 at 04:18 To: Neale Ranns <ne...@graphiant.com<mailto:ne...@graphiant.com>> Cc: vpp-dev <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> Subject: Re: [vpp-dev] GRE-over-IPSec fails Hi Neale, Could you let me know if you faced the mentioned problem anytime? For me only IPSec works fine, Only GRE also works fine. But when I configure GRE-over-IPSec, the traffic is dropped at esp4-decrypt-tun due to integrity check failure. As there are two logical interfaces created at VPP (ipip0 and gre0) for the peer, do I need to take care of something specially? As far as I know, I haven't missed any config. Regards, Vijay Kumar N On Mon, Mar 22, 2021 at 11:31 PM Vijay Kumar via lists.fd.io<http://lists.fd.io> <vjkumar2003=gmail....@lists.fd.io<mailto:gmail....@lists.fd.io>> wrote: Hi, I am trying a test case where-in I have an GRE P2MP (mGRE) tunnel on the VPP. The GRE peer is a strongswan VM that hosts both the GRE tunnel and IPSec SA. When I started ping traffic from SS, the traffic is dropped at esp4-decrypt-tun graph node due to integrity check failure. Has any one tested GRE-over-IPSec recently? If so can you pls share me a working config. If not please review the below config and let me know if I missed something NOTE: - If I have run only GRE test case, traffic is fine (no IPSec enabled). If I have only IPSec configured but no GRE then also traffic is fine. I am facing this issue only when both GRE and IPSec are enabled at the same time. Topology and config at SS and VPP ============================== Strongswan VM (20.20.99.215, gre peer 2.2.2.1, loopback 7.7.7.7) <=============> VPP cluster (20.20.99.99, gre peer 2.2.2.2, loopback 8.8.8.8) IPSec SA Traffic Selector (7.7.7.7/32<http://7.7.7.7/32> to 8.8.8.8/32<http://8.8.8.8/32>) ike=aes256-sha256-modp2048! esp=aes256-sha1-noesn! Below is the VPP trace ================ 03:20:34:670201: dpdk-input VirtualFuncEthernet0/7/0 rx queue 0 buffer 0x4c6b91: current data 0, length 170, buffer-pool 0, ref-count 1, totlen-nifb 0, trace handle 0x1000000 ext-hdr-valid l4-cksum-computed l4-cksum-correct PKT MBUF: port 0, nb_segs 1, pkt_len 170 buf_len 2176, data_len 170, ol_flags 0x180, data_off 128, phys_addr 0xa3dae4c0 packet_type 0x691 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 rss 0x0 fdir.hi 0x0 fdir.lo 0x0 Packet Offload Flags PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid Packet Types RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or without extension headers RTE_PTYPE_L4_NONFRAG (0x0600) Non-fragmented IP packet IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:c2:b4:f4 802.1q vlan 1556 IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 tos 0x00, ttl 64, length 152, checksum 0x5b33 dscp CS0 ecn NON_ECN fragment id 0xef9e, flags DONT_FRAGMENT 03:20:34:670208: ethernet-input frame: flags 0x3, hw-if-index 3, sw-if-index 3 IP4: fa:16:3e:4b:6b:42 -> fa:16:3e:c2:b4:f4 802.1q vlan 1556 03:20:34:670214: ip4-input IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 tos 0x00, ttl 64, length 152, checksum 0x5b33 dscp CS0 ecn NON_ECN fragment id 0xef9e, flags DONT_FRAGMENT 03:20:34:670218: ip4-lookup fib 1 dpo-idx 21 flow hash: 0x00000000 IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 tos 0x00, ttl 64, length 152, checksum 0x5b33 dscp CS0 ecn NON_ECN fragment id 0xef9e, flags DONT_FRAGMENT 03:20:34:670220: ip4-local IPSEC_ESP: 20.20.99.215 -> 20.20.99.99 tos 0x00, ttl 64, length 152, checksum 0x5b33 dscp CS0 ecn NON_ECN fragment id 0xef9e, flags DONT_FRAGMENT 03:20:34:670222: ipsec4-tun-input IPSec: remote:20.20.99.215 spi:305419897 (0x12345679) seq 40 sa 1 03:20:34:670225: esp4-decrypt-tun esp: crypto aes-cbc-256 integrity sha1-96 pkt-seq 40 sa-seq 0 sa-seq-hi 0 03:20:34:670241: ip4-drop IP6_NONXT: 242.163.36.86 -> 70.168.225.19 version 1, header length 8 tos 0x34, ttl 245, length 22137, checksum 0x5156 (should be 0x972a) dscp unknown ecn NON_ECN fragment id 0x0000 offset 320 03:20:34:670243: error-drop rx:ipip0 03:20:34:670244: drop esp4-decrypt-tun: Integrity check failed vpp# show node counters Count Node Reason 25 esp4-encrypt-tun ESP pkts received 213 memif-input not ip packet 3 dpdk-input no error 136 arp-reply ARP replies sent 3 arp-reply IP4 source address not local to subnet 1 gre4-input no error 213 ip4-udp-lookup No error 42 esp4-decrypt-tun ESP pkts received 42 esp4-decrypt-tun Integrity check failed 25 esp4-encrypt-tun ESP pkts received 42 ipsec4-tun-input good packets received 11 ip4-local ip4 source lookup miss 3 ip4-local unknown ip protocol 3 ethernet-input unknown vlan vpp#
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#19036): https://lists.fd.io/g/vpp-dev/message/19036 Mute This Topic: https://lists.fd.io/mt/81531694/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
