On Wed, Dec 21, 2022 at 4:08 PM Gavin McKee <gavmcke...@googlemail.com> wrote: > > Hi Frode, > > Thanks for your email and for taking the time to look at this.
You're very welcome, thank you for collecting more information. > Kernel version > > root@usc01a-032-16a:/home/gmckee# uname -r > 5.15.0-53-generic At that kernel version the `hw_addr` attribute is available, however as discussed below this does not help us for the non-DPU use case. > Output from the devlink port does not show a hw_addr on the physical port . > > root@usc01a-032-16a:/home/gmckee# devlink port > pci/0000:07:00.0/65535: type eth netdev enp7s0f0 flavour physical port 0 > splittable false > pci/0000:07:00.0/1: type eth netdev enp7s0f0_0 flavour pcivf controller 0 > pfnum 0 vfnum 0 external false splittable false > function: > hw_addr 10:70:fd:ab:cd:01 > pci/0000:07:00.0/2: type eth netdev enp7s0f0_1 flavour pcivf controller 0 > pfnum 0 vfnum 1 external false splittable false > function: > hw_addr 10:70:fd:ab:cd:02 > pci/0000:07:00.0/3: type eth netdev enp7s0f0_2 flavour pcivf controller 0 > pfnum 0 vfnum 2 external false splittable false > function: > hw_addr 10:70:fd:ab:cd:03 > pci/0000:07:00.0/4: type eth netdev enp7s0f0_3 flavour pcivf controller 0 > pfnum 0 vfnum 3 external false splittable false > function: > hw_addr 10:70:fd:ab:cd:04 > pci/0000:07:00.0/5: type eth netdev enp7s0f0_4 flavour pcivf controller 0 > pfnum 0 vfnum 4 external false splittable false > function: > hw_addr 10:70:fd:e2:44:44 > pci/0000:07:00.0/6: type eth netdev enp7s0f0_5 flavour pcivf controller 0 > pfnum 0 vfnum 5 external false splittable false > function: > hw_addr 10:70:fd:e2:a3:02 > pci/0000:07:00.1/131071: type eth netdev enp7s0f1 flavour physical port 1 > splittable false > pci/0000:07:00.3/196608: type eth netdev enp7s0f0v1 flavour virtual > splittable false > pci/0000:07:00.4/262144: type eth netdev enp7s0f0v2 flavour virtual > splittable false > > Log messages below > > root@usc01a-032-16a:/home/gmckee# ovn-controller > 2022-12-20T21:42:02Z|00001|vif_plug_representor|WARN|attempt to add function > before having knowledge about PF [ snip ] The representor plugin currently assumes the presence of PCI_PF flavoured ports and uses them to correlate to which PF each PCI_VF port belongs. When the ovn-controller runs on the SmartNIC DPU side of a PCI complex, these ports represent resources presented to the host side of the PCI complex. When using an accelerator card that exposes the physical ports and the embedded switch to the host system there will be no PCI_PF ports. The devlink-port infrastructure is still useful in this topology because it can provide a unified way of managing subfunctions [2]. I guess we could try to find a way to detect this mode of operation, or introduce an option, and then use some other method to correlate the resources. At the point in time where the message is logged [3], there is a lot of information available, so we just need to find a light weight and compatible way of doing it. 2: https://legacy.netdevconf.info/0x14/pub/papers/45/0x14-paper45-talk-paper.pdf 3: https://github.com/ovn-org/ovn-vif/blob/ce1a36f300a74b4eae55a7fec7d18da8b9218e29/lib/vif-plug-providers/representor/vif-plug-representor.c#L321-L328 -- Frode Nordahl > On Wed, 21 Dec 2022 at 02:50, Frode Nordahl <frode.nord...@canonical.com> > wrote: >> >> Hello, Gavin, >> >> Thank you for your interest in the vif plug infrastructure and the >> representor port plugin. See replies inline below. >> >> tir. 20. des. 2022, 19:29 skrev Gavin McKee via discuss >> <ovs-discuss@openvswitch.org>: >>> >>> Hi, >>> >>> We are hoping someone can help with the following error message. >>> >>> Here we add the required options to the logical switch port in OVN North >>> ovn-nbctl lsp-set-options c1-sw0-p1 requested-chassis=usc01a-032-16a >>> vif-plug-type=representor vif-plug:representor:pf-mac=10:70:fd:df:9c:3a >>> vif-plug:representor:vf-num=4 >>> >>> When I check the ovn-controller log on the hypervisor I see the following >>> error message: >>> 2022-12-20T18:24:42.815Z|00108|vif_plug|INFO|Not plugging lport c1-sw0-p1 >>> on direction from VIF plug provider. >>> 2022-12-20T18:24:47.816Z|00109|vif_plug_representor|INFO|No representor >>> port found for lport: c1-sw0-p1 pf-mac: '10:70:fd:df:9c:3a' vf-num: '4' >>> >>> Here is the information for the Mellanox Connect X6 card we are using , you >>> can see the mac on the physical interface is defined in the entry >>> vif-plug:representor:pf-mac=10:70:fd:df:9c:3a >>> ``` >>> root@usc01a-032-16a:/home/gmckee# ip link show enp7s0f0 >>> 14: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9214 qdisc mq master >>> ovs-system state UP mode DEFAULT group default qlen 1000 >>> link/ether 10:70:fd:df:9c:3a brd ff:ff:ff:ff:ff:ff >>> vf 0 link/ether 10:70:fd:ab:cd:01 brd ff:ff:ff:ff:ff:ff, spoof >>> checking off, link-state disable, trust off, query_rss off >>> vf 1 link/ether 10:70:fd:ab:cd:02 brd ff:ff:ff:ff:ff:ff, spoof >>> checking off, link-state disable, trust off, query_rss off >>> vf 2 link/ether 10:70:fd:ab:cd:03 brd ff:ff:ff:ff:ff:ff, spoof >>> checking off, link-state disable, trust off, query_rss off >>> vf 3 link/ether 10:70:fd:ab:cd:04 brd ff:ff:ff:ff:ff:ff, spoof >>> checking off, link-state disable, trust off, query_rss off >>> vf 4 link/ether 10:70:fd:e2:44:44 brd ff:ff:ff:ff:ff:ff, spoof >>> checking off, link-state disable, trust off, query_rss off >>> vf 5 link/ether 10:70:fd:e2:a3:02 brd ff:ff:ff:ff:ff:ff, spoof >>> checking off, link-state disable, trust off, query_rss off >>> altname enp7s0f0np0 >>> ``` >>> >>> The representor information is as follows >>> >>> root@usc01a-032-16a:/home/gmckee# ip link show enp7s0f0_4 >>> 32: enp7s0f0_4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state >>> UP mode DEFAULT group default qlen 1000 >>> link/ether ae:4e:85:a3:83:22 brd ff:ff:ff:ff:ff:ff >>> altname enp7s0f0npf0vf4 >>> >>> >>> the virtual function has already been assigned to a KVM VM. >>> >>> Any help is greatly appreciated . >> >> >> The representor plugin was developed for and tested with a SmartNIC DPU, >> which behaves slightly different than a system where the embedded switch is >> exposed to the host system. >> >> Having said that, it was developed using generic interfaces, such as >> devlink-port [0], so we should be able to make it work. >> >> The representor plugin looks up the representor by combining information >> about PF MAC (`hw_addr`) and VF number from devlink [2], a recent kernel >> version is required to expose the `hw_addr` attribute. >> >> A few questions: >> Do you see any other messages logged from the vif_plug_representor module? >> >> What kernel version is in use? >> >> Does the `hw_addr` show up for the PCI_PF flavoured port in `devlink port >> show`? >> >> 0: >> https://www.kernel.org/doc/html/latest/networking/devlink/devlink-port.html >> 1: >> https://github.com/ovn-org/ovn-vif/blob/ce1a36f300a74b4eae55a7fec7d18da8b9218e29/lib/vif-plug-providers/representor/vif-plug-representor.c#L407-L469 >> >> -- >> Frode Nordahl >> >>> >>> Gav >>> >>> _______________________________________________ >>> discuss mailing list >>> disc...@openvswitch.org >>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
