On Thu, May 25, 2023 at 9:03 AM Robert Navarro <crsh...@gmail.com> wrote: > > Hi Frode, > > Thanks for the fast reply! > > Replies in-line as well. > > On Wed, May 24, 2023 at 11:41 PM Frode Nordahl <frode.nord...@canonical.com> > wrote: >> >> Hello, Robert, >> >> See my response in-line below. >> >> On Thu, May 25, 2023 at 8:20 AM Robert Navarro via discuss >> <ovs-discuss@openvswitch.org> wrote: >> > >> > Hello, >> > >> > I've followed the directions here: >> > https://docs.nvidia.com/networking/pages/viewpage.action?pageId=119763689 >> > >> > But I can't seem to get HW offload to work on my system. >> > >> > I'm using the latest OFED drivers with a ConnectX-5 SFP28 card running on >> > kernel 5.15.107-2-pve >> >> Note that if you plan to use this feature with OVN you may find that >> the ConnectX-5 does not provide all the features required. Among other >> things it does not support the `dec_ttl` action which is a >> prerequisite for processing L3 routing, I'm also not sure whether it >> fully supports connection tracking offload. I'd go with CX-6 DX or >> above if this is one of your use cases. > > Good to know, I'll keep that in mind as I progress with testing. > >> >> >> > I have two hosts directly connected to each other, running a simple ping >> > between hosts shows the following flows: >> > >> > root@pvet1:~# ovs-appctl dpctl/dump-flows -m type=tc >> > ufid:be1670c1-b36b-4f0f-8aba-e9415b9d0fb1, >> > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens15f0np0),packet_type(ns=0/0,id=0/0),eth(src=6e:56:fd:40:6e:22,dst=f6:36:11:c6:04:f0),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), >> > packets:639, bytes:53676, used:0.710s, dp:tc, actions:tap103i1 >> > >> > root@pvet2:~# ovs-appctl dpctl/dump-flows -m type=tc >> > ufid:f4d0ebd2-7ba9-4e21-9bf8-090f90bac072, >> > skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(ens15f0np0),packet_type(ns=0/0,id=0/0),eth(src=f6:36:11:c6:04:f0,dst=6e:56:fd:40:6e:22),eth_type(0x0800),ipv4(src=0.0.0.0/0.0.0.0,dst=0.0.0.0/0.0.0.0,proto=0/0,tos=0/0,ttl=0/0,frag=no), >> > packets:656, bytes:55104, used:0.390s, dp:tc, actions:tap100i1 >> >> The two flows listed above have an action towards what looks like a >> virtual Ethernet tap interface. That is not a supported configuration >> for hardware offload. > > I'm using Proxmox as the hypervisor. > > It seems like Proxmox is attaching the VM (using a virtio nic) as a tap > (tap100i1) to the OVS: > > root@pvet2:~# ovs-dpctl show > system@ovs-system: > lookups: hit:143640674 missed:1222 lost:45 > flows: 2 > masks: hit:143642152 total:2 hit/pkt:1.00 > port 0: ovs-system (internal) > port 1: vmbr1 (internal) > port 2: ens15f0np0 > port 3: vlan44 (internal) > port 4: vlan66 (internal) > port 5: ens15f0npf0vf0 > port 6: ens15f0npf0vf1 > port 7: tap100i1 > > Given that proxmox uses KVM for virtualization, what's the correct way to > link a KVM VM to OVS?
I do not have detailed knowledge about proxmox, but what you would be looking for is PCI Passthrough and SR-IOV. Once you get the instance to use a VF instead of a virtio nic you should add its representor port to the OVS bridge. You can find the names of the representor ports by issuing the `devlink port show` command. >> >> The instance needs to be connected to an > > When you say instance here, do you mean the KVM virtual machine or the > instance of OVS? Instance as in the KVM virtual machine instance. >> >> interface wired directly to the embedded switch in the card by >> attaching a VF or SF to the instance. > > OVS is attached to the physical nic using the PF and 2 VFs as shown in the > ovs-dpctl output above Beware that there are multiple types of representor ports in play here and you would be interested in plugging the ports with flavour `virtual` into the OVS bridge. -- Frode Nordahl >> >> >> -- >> Frode Nordahl >> >> > Neither of which shows offloaded >> > >> > The commands I'm using to setup the interfaces are: >> > >> > echo 2 | tee /sys/class/net/ens15f0np0/device/sriov_numvfs >> > >> > lspci -nn | grep Mellanox >> > >> > echo 0000:03:00.2 | tee /sys/bus/pci/drivers/mlx5_core/unbind >> > echo 0000:03:00.3 | tee /sys/bus/pci/drivers/mlx5_core/unbind >> > >> > devlink dev eswitch set pci/0000:03:00.0 mode switchdev >> > >> > echo 0000:03:00.2 | tee /sys/bus/pci/drivers/mlx5_core/bind >> > echo 0000:03:00.3 | tee /sys/bus/pci/drivers/mlx5_core/bind >> > >> > ovs-vsctl set Open_vSwitch . other_config:hw-offload=true >> > >> > systemctl restart openvswitch-switch.service >> > >> > ovs-vsctl add-port vmbr1 ens15f0np0 >> > ovs-vsctl add-port vmbr1 ens15f0npf0vf0 >> > ovs-vsctl add-port vmbr1 ens15f0npf0vf1 >> > >> > ethtool -K ens15f0np0 hw-tc-offload on >> > ethtool -K ens15f0npf0vf0 hw-tc-offload on >> > ethtool -K ens15f0npf0vf1 hw-tc-offload on >> > >> > ip link set dev ens15f0np0 up >> > ip link set dev ens15f0npf0vf0 up >> > ip link set dev ens15f0npf0vf1 up >> > >> > ovs-dpctl show >> > >> > Any ideas on what else to check? >> > >> > -- >> > Robert Navarro >> > _______________________________________________ >> > discuss mailing list >> > disc...@openvswitch.org >> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss > > > > -- > Robert Navarro _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
