** Description changed: Summary: Align Kernel IPsec Full offload implementation in the DPU to the upstream Full offload in all components: OFED, Strongswan, etc. This is in order for DPU Kernel IPsec to include policy offload and be fully aligned to what CX Kernel customers will use. How to test: - Host 1: + Host 1 + Enable sriov and set namespace. + + ip link set eth2 up + echo '1' > /sys/class/net/eth2/device/sriov_numvfs + ip netns add nt1 + ip link set eth4 netns nt1 + ip netns exec nt1 ifconfig eth4 11.11.11.1/24 up + + BF on host 1: + Set steering mode to "dmfs". By default, it is "smfs" and not supported for now. + /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev - - BF on host 1: - /opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir out tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0xefa83812 mode transport priority 10 - /opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir in tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0x63a7db74 mode transport priority 10 - /opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir fwd tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0x63a7db74 mode transport priority 10 - /opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165/16 dst 196.234.182.166/16 proto esp spi 0xefa83812 reqid 0xefa83812 mode transport aead 'rfc4106(gcm(aes))' 0xe2fe3857301d8f72b5d71d295a462ef21868e407 128 offload packet dev p0 dir out sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32 - /opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.182.166/16 dst 196.234.181.165/16 proto esp spi 0x63a7db74 reqid 0x63a7db74 mode transport aead 'rfc4106(gcm(aes))' 0xe916c4d0db1886e8c877b023e8cebef53b4d2d0f 128 offload packet dev p0 dir in sel src 196.234.182.166/16 dst 196.234.181.165/16 flag esn replay-window 32 - - Start OVS and set following configure on BF: - /usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true - /usr/bin/ovs-vsctl set Open_vSwitch . other_config:max-idle=300000 - - Host2: + /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev - BF on host 2: - /opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.182.166 dst 196.234.181.165 dir out tmpl src 196.234.182.166/16 dst 196.234.181.165/16 proto esp reqid 0xefa83812 mode transport priority 10 - /opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir in tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0x63a7db74 mode transport priority 10 - /opt/mellanox/iproute2/sbin/ip xfrm policy add src 196.234.181.165 dst 196.234.182.166 dir fwd tmpl src 196.234.181.165/16 dst 196.234.182.166/16 proto esp reqid 0x63a7db74 mode transport priority 10 - /opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165 dst 196.234.182.166 proto esp spi 0xefa83812 reqid 0xefa83812 mode transport aead 'rfc4106(gcm(aes))' 0xe2fe3857301d8f72b5d71d295a462ef21868e407 128 offload packet dev p0 dir out sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32 - /opt/mellanox/iproute2/sbin/ip xfrm state add src 196.234.181.165 dst 196.234.182.166 proto esp spi 0x63a7db74 reqid 0x63a7db74 mode transport aead 'rfc4106(gcm(aes))' 0xe916c4d0db1886e8c877b023e8cebef53b4d2d0f 128 offload packet dev p0 dir in sel src 196.234.181.165/16 dst 196.234.182.166/16 flag esn replay-window 32 + IPSec configure + /opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.2 dst 2.2.2.3 offload packet dev p0 dir out tmpl src 2.2.2.2/16 dst 2.2.2.3/16 proto esp reqid 0xb29ed314 mode transport priority 12 + /opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.3 dst 2.2.2.2 offload packet dev p0 dir in tmpl src 2.2.2.3/16 dst 2.2.2.2/16 proto esp reqid 0xc35aa26e mode transport priority 12 + /opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.2/16 dst 2.2.2.3/16 proto esp spi 0xb29ed314 reqid 0xb29ed314 mode transport aead 'rfc4106(gcm(aes))' 0x20f01f80a26f633d85617465686c32552c92c42f 128 offload packet dev p0 dir out sel src 2.2.2.2/16 dst 2.2.2.3/16 flag esn replay-window 64 + /opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.3/16 dst 2.2.2.2/16 proto esp spi 0xc35aa26e reqid 0xc35aa26e mode transport aead 'rfc4106(gcm(aes))' 0x6cb228189b4c6e82e66e46920a2cde39187de4ba 128 offload packet dev p0 dir in sel src 2.2.2.3/16 dst 2.2.2.2/16 flag esn replay-window 64 + OVS configure. Clear all bridges before configure if there's already default bridges in BF. - Start OVS and set following configure on BF: - /usr/bin/ovs-vsctl set Open_vSwitch . other_config:hw-offload=true - /usr/bin/ovs-vsctl set Open_vSwitch . other_config:max-idle=300000 + ovs-vsctl set Open_vSwitch . other_config:hw-offload=false # need to restart ovs after setting this command + + ovs-vsctl add-br br-int + ovs-vsctl add-port br-int pf0vf0 -- set interface pf0vf0 options:representor=[0] + ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan options:key=100 options:local_ip=2.2.2.2 options:remote_ip=2.2.2.3 options:dst_port=4789 + Configure IP - Send the traffic between host 1 and host 2 and check IPsec counters in - "ethtool -S" statistics on both BF. + ifconfig p0 2.2.2.2/16 up + + + Host2: + Enable sriov and set namespace. + + ip link set eth2 up + echo '1' > /sys/class/net/eth2/device/sriov_numvfs + ip netns add nt1 + ip link set eth4 netns nt1 + ip netns exec nt1 ifconfig eth4 11.11.11.2/24 up + + BF on host 2 + Set steering mode + + /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy + echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode + echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode + /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev + + /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy + echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode + echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode + /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev + IPSec configure + + /opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.3 dst 2.2.2.2 offload packet dev p0 dir out tmpl src 2.2.2.3/16 dst 2.2.2.2/16 proto esp reqid 0xc35aa26e mode transport priority 12 + /opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.2 dst 2.2.2.3 offload packet dev p0 dir in tmpl src 2.2.2.2/16 dst 2.2.2.3/16 proto esp reqid 0xb29ed314 mode transport priority 12 + /opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.3/16 dst 2.2.2.2/16 proto esp spi 0xc35aa26e reqid 0xc35aa26e mode transport aead 'rfc4106(gcm(aes))' 0x6cb228189b4c6e82e66e46920a2cde39187de4ba 128 offload packet dev p0 dir out sel src 2.2.2.3/16 dst 2.2.2.2/16 flag esn replay-window 64 + /opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.2/16 dst 2.2.2.3/16 proto esp spi 0xb29ed314 reqid 0xb29ed314 mode transport aead 'rfc4106(gcm(aes))' 0x20f01f80a26f633d85617465686c32552c92c42f 128 offload packet dev p0 dir in sel src 2.2.2.2/16 dst 2.2.2.3/16 flag esn replay-window 64 + OVS configure + + ovs-vsctl set Open_vSwitch . other_config:hw-offload=false # need to restart ovs after setting this command + + ovs-vsctl add-br br-int + ovs-vsctl add-port br-int pf0vf0 -- set interface pf0vf0 options:representor=[0] + ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan options:key=100 options:local_ip=2.2.2.3 options:remote_ip=2.2.2.2 options:dst_port=4789 + Configure IP + + ifconfig p0 2.2.2.3/16 up + + On host 2, check ping 11.11.11.1 is OK and check IPsec counters in + "ethtool -S p0" statistics on both BF. + + root@c-237-153-60-065:~# ip netns exec nt1 ping 11.11.11.1 How to fix: Need to backport a series of xfrm patches into BlueField 5.15 kernel, from 6.0 upstream kernel. Patches needed for 5.15 kernel: afe9e47 xfrm: fix conflict for netdev and tx stats 6aff54d xfrm: don't skip free of empty state in acquire policy 692fecb xfrm: delete offloaded policy 91b6276 xfrm: Support UDP encapsulation in packet offload mode 69e168a xfrm: add missed call to delete offloaded policies 9724724 xfrm: release all offloaded policy memory e57b7ec xfrm: don't require advance ESN callback for packet offload 9e98488 xfrm: copy_to_user_state fetch offloaded SA packets/bytes statistics 4778c10 xfrm: add new device offload acquire flag 2601c94 netlink: provide an ability to set default extack message b4951d5 netlink: add support for formatted extack messages b5dd0fa xfrm: extend add state callback to set failure reason 326a004 xfrm: extend add policy callback to set failure reason 40b173d1 xfrm: document IPsec packet offload mode b1737ae xfrm: add support to HW update soft and hard limits cad4cd7 xfrm: speed-up lookup of HW policies b347fe7 xfrm: add TX datapath support for IPsec packet offload mode cfcc50f xfrm: add an interface to offload policy 2f7e5f7 xfrm: propagate extack to all netlink doit handlers 8d459bb xfrm: add extack to verify_policy_type 3563725 xfrm: allow state packet offload mode 207abea xfrm: add extack support to xfrm_dev_state_add facf282 xfrm: add new packet offload flag 6f12533 xfrm: Remove not-used total variable 46bd9eb xfrm: drop not needed flags variable in XFRM offload struct bbadbe7 xfrm: store and rely on direction to construct offload flags c01b278 xfrm: rename xfrm_state_offload struct to allow reuse f337706 xfrm: delete not used number of external headers db0cee8 Revert "UBUNTU: SAUCE: net/xfrm: Fix XFRM flags validity check" 57995bb Revert "UBUNTU: SAUCE: net/xfrm: IPsec full offload support for lifetime limit" 244050a Revert "UBUNTU: SAUCE: net/xfrm: Add support for xfrm full offload"
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-bluefield in Ubuntu. https://bugs.launchpad.net/bugs/2034578 Title: Support IPSEC full offload implementation Status in linux-bluefield package in Ubuntu: Invalid Status in linux-bluefield source package in Jammy: Fix Committed Bug description: Summary: Align Kernel IPsec Full offload implementation in the DPU to the upstream Full offload in all components: OFED, Strongswan, etc. This is in order for DPU Kernel IPsec to include policy offload and be fully aligned to what CX Kernel customers will use. How to test: Host 1 Enable sriov and set namespace. ip link set eth2 up echo '1' > /sys/class/net/eth2/device/sriov_numvfs ip netns add nt1 ip link set eth4 netns nt1 ip netns exec nt1 ifconfig eth4 11.11.11.1/24 up BF on host 1: Set steering mode to "dmfs". By default, it is "smfs" and not supported for now. /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev IPSec configure /opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.2 dst 2.2.2.3 offload packet dev p0 dir out tmpl src 2.2.2.2/16 dst 2.2.2.3/16 proto esp reqid 0xb29ed314 mode transport priority 12 /opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.3 dst 2.2.2.2 offload packet dev p0 dir in tmpl src 2.2.2.3/16 dst 2.2.2.2/16 proto esp reqid 0xc35aa26e mode transport priority 12 /opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.2/16 dst 2.2.2.3/16 proto esp spi 0xb29ed314 reqid 0xb29ed314 mode transport aead 'rfc4106(gcm(aes))' 0x20f01f80a26f633d85617465686c32552c92c42f 128 offload packet dev p0 dir out sel src 2.2.2.2/16 dst 2.2.2.3/16 flag esn replay-window 64 /opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.3/16 dst 2.2.2.2/16 proto esp spi 0xc35aa26e reqid 0xc35aa26e mode transport aead 'rfc4106(gcm(aes))' 0x6cb228189b4c6e82e66e46920a2cde39187de4ba 128 offload packet dev p0 dir in sel src 2.2.2.3/16 dst 2.2.2.2/16 flag esn replay-window 64 OVS configure. Clear all bridges before configure if there's already default bridges in BF. ovs-vsctl set Open_vSwitch . other_config:hw-offload=false # need to restart ovs after setting this command ovs-vsctl add-br br-int ovs-vsctl add-port br-int pf0vf0 -- set interface pf0vf0 options:representor=[0] ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan options:key=100 options:local_ip=2.2.2.2 options:remote_ip=2.2.2.3 options:dst_port=4789 Configure IP ifconfig p0 2.2.2.2/16 up Host2: Enable sriov and set namespace. ip link set eth2 up echo '1' > /sys/class/net/eth2/device/sriov_numvfs ip netns add nt1 ip link set eth4 netns nt1 ip netns exec nt1 ifconfig eth4 11.11.11.2/24 up BF on host 2 Set steering mode /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode legacy echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.0/net/p0/compat/devlink/steering_mode echo 'full' > /sys/class/net/p0/compat/devlink/ipsec_mode /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.0 mode switchdev /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode legacy echo 'dmfs' > /sys/bus/pci/devices/0000:03:00.1/net/p1/compat/devlink/steering_mode echo 'full' > /sys/class/net/p1/compat/devlink/ipsec_mode /opt/mellanox/iproute2/sbin/devlink dev eswitch set pci/0000:03:00.1 mode switchdev IPSec configure /opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.3 dst 2.2.2.2 offload packet dev p0 dir out tmpl src 2.2.2.3/16 dst 2.2.2.2/16 proto esp reqid 0xc35aa26e mode transport priority 12 /opt/mellanox/iproute2/sbin/ip xfrm policy add src 2.2.2.2 dst 2.2.2.3 offload packet dev p0 dir in tmpl src 2.2.2.2/16 dst 2.2.2.3/16 proto esp reqid 0xb29ed314 mode transport priority 12 /opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.3/16 dst 2.2.2.2/16 proto esp spi 0xc35aa26e reqid 0xc35aa26e mode transport aead 'rfc4106(gcm(aes))' 0x6cb228189b4c6e82e66e46920a2cde39187de4ba 128 offload packet dev p0 dir out sel src 2.2.2.3/16 dst 2.2.2.2/16 flag esn replay-window 64 /opt/mellanox/iproute2/sbin/ip xfrm state add src 2.2.2.2/16 dst 2.2.2.3/16 proto esp spi 0xb29ed314 reqid 0xb29ed314 mode transport aead 'rfc4106(gcm(aes))' 0x20f01f80a26f633d85617465686c32552c92c42f 128 offload packet dev p0 dir in sel src 2.2.2.2/16 dst 2.2.2.3/16 flag esn replay-window 64 OVS configure ovs-vsctl set Open_vSwitch . other_config:hw-offload=false # need to restart ovs after setting this command ovs-vsctl add-br br-int ovs-vsctl add-port br-int pf0vf0 -- set interface pf0vf0 options:representor=[0] ovs-vsctl add-port br-int vxlan0 -- set interface vxlan0 type=vxlan options:key=100 options:local_ip=2.2.2.3 options:remote_ip=2.2.2.2 options:dst_port=4789 Configure IP ifconfig p0 2.2.2.3/16 up On host 2, check ping 11.11.11.1 is OK and check IPsec counters in "ethtool -S p0" statistics on both BF. root@c-237-153-60-065:~# ip netns exec nt1 ping 11.11.11.1 How to fix: Need to backport a series of xfrm patches into BlueField 5.15 kernel, from 6.0 upstream kernel. Patches needed for 5.15 kernel: afe9e47 xfrm: fix conflict for netdev and tx stats 6aff54d xfrm: don't skip free of empty state in acquire policy 692fecb xfrm: delete offloaded policy 91b6276 xfrm: Support UDP encapsulation in packet offload mode 69e168a xfrm: add missed call to delete offloaded policies 9724724 xfrm: release all offloaded policy memory e57b7ec xfrm: don't require advance ESN callback for packet offload 9e98488 xfrm: copy_to_user_state fetch offloaded SA packets/bytes statistics 4778c10 xfrm: add new device offload acquire flag 2601c94 netlink: provide an ability to set default extack message b4951d5 netlink: add support for formatted extack messages b5dd0fa xfrm: extend add state callback to set failure reason 326a004 xfrm: extend add policy callback to set failure reason 40b173d1 xfrm: document IPsec packet offload mode b1737ae xfrm: add support to HW update soft and hard limits cad4cd7 xfrm: speed-up lookup of HW policies b347fe7 xfrm: add TX datapath support for IPsec packet offload mode cfcc50f xfrm: add an interface to offload policy 2f7e5f7 xfrm: propagate extack to all netlink doit handlers 8d459bb xfrm: add extack to verify_policy_type 3563725 xfrm: allow state packet offload mode 207abea xfrm: add extack support to xfrm_dev_state_add facf282 xfrm: add new packet offload flag 6f12533 xfrm: Remove not-used total variable 46bd9eb xfrm: drop not needed flags variable in XFRM offload struct bbadbe7 xfrm: store and rely on direction to construct offload flags c01b278 xfrm: rename xfrm_state_offload struct to allow reuse f337706 xfrm: delete not used number of external headers db0cee8 Revert "UBUNTU: SAUCE: net/xfrm: Fix XFRM flags validity check" 57995bb Revert "UBUNTU: SAUCE: net/xfrm: IPsec full offload support for lifetime limit" 244050a Revert "UBUNTU: SAUCE: net/xfrm: Add support for xfrm full offload" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-bluefield/+bug/2034578/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
