> -----Original Message----- > From: Samudrala, Sridhar <sridhar.samudr...@intel.com> > Sent: Friday, March 15, 2019 12:58 AM > To: Parav Pandit <pa...@mellanox.com>; Jakub Kicinski > <jakub.kicin...@netronome.com> > Cc: Jiri Pirko <j...@resnulli.us>; da...@davemloft.net; > netdev@vger.kernel.org; oss-driv...@netronome.com > Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI > ports > > > On 3/14/2019 7:40 PM, Parav Pandit wrote: > > > > > >> -----Original Message----- > >> From: Samudrala, Sridhar <sridhar.samudr...@intel.com> > >> Sent: Thursday, March 14, 2019 9:16 PM > >> To: Parav Pandit <pa...@mellanox.com>; Jakub Kicinski > >> <jakub.kicin...@netronome.com> > >> Cc: Jiri Pirko <j...@resnulli.us>; da...@davemloft.net; > >> netdev@vger.kernel.org; oss-driv...@netronome.com > >> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on > >> devlink PCI ports > >> > >> > >> > >> On 3/14/2019 6:28 PM, Parav Pandit wrote: > >>> > >>> > >>>> -----Original Message----- > >>>> From: Jakub Kicinski <jakub.kicin...@netronome.com> > >>>> Sent: Thursday, March 14, 2019 6:39 PM > >>>> To: Parav Pandit <pa...@mellanox.com> > >>>> Cc: Jiri Pirko <j...@resnulli.us>; da...@davemloft.net; > >>>> netdev@vger.kernel.org; oss-driv...@netronome.com > >>>> Subject: Re: [PATCH net-next v2 4/7] devlink: allow subports on > >>>> devlink PCI ports > >>>> > >>>> On Thu, 14 Mar 2019 22:35:36 +0000, Parav Pandit wrote: > >>>>>>> Then instances of flavour pci_vf are going to appear in the same > >>>>>>> devlink instance. Those are the switch ports: > >>>>>>> pci/0000:05:00.0/10002: type eth netdev enp5s0npf0pf0s0 > >>>>>>> flavour pci_vf pf 0 vf 0 > >>>>>>> switch_id 00154d130d2f peer > >>>>>>> pci/0000:05:10.1/0 > >>>>>>> pci/0000:05:00.0/10003: type eth netdev enp5s0npf0pf0s0 > >>>>>>> flavour pci_vf pf 0 vf 0 subport 1 > >>>>>>> switch_id 00154d130d2f peer > >>>>>>> pci/0000:05:10.1/1 > >>>>>>> > >>>>>>> With that, peers are going to appear too, and those are the > >>>>>>> actual VF/VF > >>>>>>> subport: > >>>>>>> pci/0000:05:10.1/0: type eth netdev ??? flavour pci_vf_host > >>>>>>> peer pci/0000:05:00.0/10002 > >>>>>>> pci/0000:05:10.1/1: type eth netdev ??? flavour pci_vf_host > >>>>>>> peer pci/0000:05:00.0/10003 > >>>>>>> > >>>>>>> Later you can push this VF along with all subports to VM. So in > >>>>>>> VM, you are going to see the VF like this: > >>>>>>> $ devlink dev > >>>>>>> pci/0000:00:08.0 > >>>>>>> $ devlink port > >>>>>>> pci/0000:00:08.0/0: type eth netdev ??? flavour pci_vf_host > >>>>>>> pci/0000:00:08.0/1: type eth netdev ??? flavour pci_vf_host > >>>>>>> > >>>>>>> And back to your question of how are they connected in eswitch. > >>>>>>> That is totally up to the original user John who did the creation. > >>>>>>> He is in charge of the eswitch on baremetal, he would configure > >>>>>>> the forwarding however he likes. > >>>>>> > >>>>>> Ack, so I think you're saying VM has to communicate to the cloud > >>>>>> environment to have this provisioned using some service API, not > >>>>>> a kernel API. That's what I wanted to confirm. > >>>>>> > >>>>>> I don't see any benefit to having the "host ports" under devlink, > >>>>>> as such I think it's a matter of preference. > >>>>> > >>>>> We need 'host ports' to configure parameters of this host port > >>>>> which is not exposed by the rep-netdev. > >>>>> Such as mac address. > >>>> > >>>> Please look at the quote of what Jiri wrote above - the host port > >>>> gets passed to the VM, you can't use it as a handle to set the MAC. > >>>> > >>>> The way to set the MAC remains: > >>>> > >>>> # devlink port set pci/0000:05:00.0/10002 peer mac_addr > >>>> 00:11:22:33:44:55 > >>>> > >>> Even though it can be done, I think this is wrong model to program > >> hostport mac address using eswitch port. > >>> All devlink objects are control objects, so what is passed to VM is > >>> what is > >> represented by devlink. > >>> VF in the VM will anyway create its devlink object. > >>> What is wrong in programming hostport? > >>> It gives a very clear view to users of topology and objects. > >> > >> The VF or any subport MAC address should be configured by the > >> orchestration layer that is running on the hypervisor and when a VF > >> is assigned to a VF, the host port is not visible to the hypervisor. > > What prevents creation of hostport due to which is not visible? > > Hostport is control port to program host side of parameters. > > It should be created when user wants to program the parameters. > > > > Model is really straight forward. > > Program host port params using hostport object. > > Program switchport params using rep-netdev. > > IIUC, Jiri/Jakub are proposing creation of 2 devlink objects for each port - > host facing ports and switch facing ports. This is in addition to the netdevs > that are created today. > I am not proposing any different. I am proposing only two changes. 1. control hostport params via referring hostport (not via indirect peer) 2. flavour should not be vf/pf, flavour should be hostport, switchport. Because switch is flat and agnostic of pf/vf/mdev.
> Are you suggesting that all the devlink objects should be visible only at the > hypervisor layer? > Of course not. Ports and params controlled by hypervisor should be exposed at hypervisor/eswitch wherever its parent devlink instance exist. Ports which should be visible inside a VM should be exposed inside a VM. So for a given VF, If eswitch is at hypervisor level, $ devlink port show pci/0000:05:00.0/10002 eth netdev flavour switchport switch_id 00154d130d2f peer pci/0000:05:10.1/0 pci/0000:05:10.1/0 eth netdev flavour hostport switch_id 00154d130d2f peer pci/0000:05:00.0/10002 where VF is enumerated, $ devlink port show pci/0000:05:10.1/0 eth netdev flavour hostport This is because unprivileged VF doesn't have visibility to eswitch and its links. > I think the terminology need to be defined clearly so that we are all on the > same page. > > > > >> Currently we have ndo_set_vf_mac_addr api that works with PF netdev, > >> but i think we are trying to move away from that API and do all the > >> configuration via the port representor netdevs. > > This is fine rep-netdev represents eswitch port. > > You normally don't go to switch to program host port params. > > > >> As the mac address cannot be configured using this netdev, i think > >> Jakub is suggesting creating a devlink opject for each port > >> representor and use that interface to set peer mac address. > > > > I understand but is convoluted interface. > > When you program host NIC mac address you talk to iLo or BIOS. > > When you program switch side mac address, you go switch/router/modem. > > > > Also programming host params on host side, also doesn't make > assumption that its connected to eswitch. > > It also doesn't assume that same connectivity for its life. > > > > If you model around how physical devices are configured, it will almost > never go wrong and still provides same level of flexibility. > > > >> We should be able use this to configure port vlan too. > >> > >> Also, instead of subport, can we call vport and support different > >> types of vports - sr-iov, siov, vmdq etc. > >> > > At switch level there are just ports. > > sriov, siov, mdev, vmdq are their couter part (peer) where it is connected. > > > >>> > >>> Also eswitch is flat. There is no need of pf/vf flavour for port. > >>> It doesn't make sense to define 'mdev' flavour which we are already > >> working. > >>> At eswitch level it is just a port, it happen to be connected to vf > >>> or pf or > >> other objects, it doesn't matter. > >>> Port should be flavoured as 'hostport' or 'switchport'. > >>> > >>> > >>>> (using the port ids from above)
