Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Alexander Duyck
On Fri, Dec 18, 2020 at 12:18 PM Jason Gunthorpe wrote: > > On Fri, Dec 18, 2020 at 11:22:12AM -0800, Alexander Duyck wrote: > > > Also as far as the patch count complaints I have seen in a few threads > > I would be fine with splitting things up so that the devlink and aux > > device creation get

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Jason Gunthorpe
On Fri, Dec 18, 2020 at 11:22:12AM -0800, Alexander Duyck wrote: > Also as far as the patch count complaints I have seen in a few threads > I would be fine with splitting things up so that the devlink and aux > device creation get handled in one set, and then we work out the > details of mlx5 atta

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Alexander Duyck
On Fri, Dec 18, 2020 at 10:01 AM Parav Pandit wrote: > > > > From: Alexander Duyck > > Sent: Friday, December 18, 2020 9:31 PM > > > > On Thu, Dec 17, 2020 at 9:20 PM Parav Pandit wrote: > > > > > > > > > > From: Alexander Duyck > > > > Sent: Friday, December 18, 2020 8:41 AM > > > > > > > > On

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Parav Pandit
> From: Alexander Duyck > Sent: Friday, December 18, 2020 9:31 PM > > On Thu, Dec 17, 2020 at 9:20 PM Parav Pandit wrote: > > > > > > > From: Alexander Duyck > > > Sent: Friday, December 18, 2020 8:41 AM > > > > > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern > wrote: > > > > > > > > On 12/16

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Alexander Duyck
On Thu, Dec 17, 2020 at 9:20 PM Parav Pandit wrote: > > > > From: Alexander Duyck > > Sent: Friday, December 18, 2020 8:41 AM > > > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > > > > > > On 12/16/20 3:53 PM, Alexander Duyck wrote: > > The problem is PCIe DMA wasn't designed to function

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-18 Thread Alexander Duyck
On Thu, Dec 17, 2020 at 7:55 PM David Ahern wrote: > > On 12/17/20 8:11 PM, Alexander Duyck wrote: > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > >> > >> On 12/16/20 3:53 PM, Alexander Duyck wrote: > >>> The problem in my case was based on a past experience where east-west > >>> traffic

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Parav Pandit
> From: Parav Pandit > Sent: Friday, December 18, 2020 10:51 AM > > > From: Alexander Duyck > > Sent: Friday, December 18, 2020 8:41 AM > > > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > > > > > > On 12/16/20 3:53 PM, Alexander Duyck wrote: > > The problem is PCIe DMA wasn't designe

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Parav Pandit
> From: Alexander Duyck > Sent: Friday, December 18, 2020 8:41 AM > > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > > > > On 12/16/20 3:53 PM, Alexander Duyck wrote: > The problem is PCIe DMA wasn't designed to function as a network switch > fabric and when we start talking about a 400Gb

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread David Ahern
On 12/17/20 8:11 PM, Alexander Duyck wrote: > On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: >> >> On 12/16/20 3:53 PM, Alexander Duyck wrote: >>> The problem in my case was based on a past experience where east-west >>> traffic became a problem and it was easily shown that bypassing the >>> N

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Alexander Duyck
On Thu, Dec 17, 2020 at 5:30 PM David Ahern wrote: > > On 12/16/20 3:53 PM, Alexander Duyck wrote: > > The problem in my case was based on a past experience where east-west > > traffic became a problem and it was easily shown that bypassing the > > NIC for traffic was significantly faster. > > If

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread David Ahern
On 12/16/20 3:53 PM, Alexander Duyck wrote: > The problem in my case was based on a past experience where east-west > traffic became a problem and it was easily shown that bypassing the > NIC for traffic was significantly faster. If a deployment expects a lot of east-west traffic *within a host* w

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Jason Gunthorpe
On Thu, Dec 17, 2020 at 01:05:03PM -0800, Alexander Duyck wrote: > > I view the SW bypass path you are talking about similarly to > > GSO/etc. It should be accessed by the HW driver as an optional service > > provided by the core netdev, not implemented as some wrapper netdev > > around a HW imple

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Alexander Duyck
On Thu, Dec 17, 2020 at 11:40 AM Jason Gunthorpe wrote: > > On Thu, Dec 17, 2020 at 10:48:48AM -0800, Alexander Duyck wrote: > > > Just to clarify I am not with Intel, nor do I plan to work on any > > Intel drivers related to this. > > Sure > > > I disagree here. In my mind a design where two inte

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Jason Gunthorpe
On Thu, Dec 17, 2020 at 10:48:48AM -0800, Alexander Duyck wrote: > Just to clarify I am not with Intel, nor do I plan to work on any > Intel drivers related to this. Sure > I disagree here. In my mind a design where two interfaces, which both > exist in the kernel, have to go to hardware in ord

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-17 Thread Alexander Duyck
On Wed, Dec 16, 2020 at 4:38 PM Jason Gunthorpe wrote: > > On Wed, Dec 16, 2020 at 02:53:07PM -0800, Alexander Duyck wrote: > > > It isn't about the association, it is about who is handling the > > traffic. Going back to the macvlan model what we did is we had a group > > of rings on the device th

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Jason Gunthorpe
On Wed, Dec 16, 2020 at 02:53:07PM -0800, Alexander Duyck wrote: > It isn't about the association, it is about who is handling the > traffic. Going back to the macvlan model what we did is we had a group > of rings on the device that would automatically forward unicast > packets to the macvlan in

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Alexander Duyck
On Wed, Dec 16, 2020 at 12:35 PM Jason Gunthorpe wrote: > > On Wed, Dec 16, 2020 at 11:27:32AM -0800, Alexander Duyck wrote: > > > That has been the case for a long time. However it had been my > > experience that SR-IOV never scaled well to meet those needs and so it > > hadn't been used in such

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Jason Gunthorpe
On Wed, Dec 16, 2020 at 11:27:32AM -0800, Alexander Duyck wrote: > That has been the case for a long time. However it had been my > experience that SR-IOV never scaled well to meet those needs and so it > hadn't been used in such deployments. Seems to be going quite well here, perhaps the applica

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Alexander Duyck
On Wed, Dec 16, 2020 at 9:51 AM Jason Gunthorpe wrote: > > On Wed, Dec 16, 2020 at 08:31:44AM -0800, Alexander Duyck wrote: > > > You say this will scale better but I am not even sure about that. The > > fact is SR-IOV could scale to 256 VFs, but for networking I kind of > > doubt the limitation w

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Saeed Mahameed
On Wed, 2020-12-16 at 08:50 +0200, Leon Romanovsky wrote: > On Tue, Dec 15, 2020 at 01:28:05PM -0800, Jakub Kicinski wrote: > > On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote: > > > > I think the big thing we really should do if we are going to go > > > > this > > > > route is to look at s

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Jason Gunthorpe
On Wed, Dec 16, 2020 at 08:31:44AM -0800, Alexander Duyck wrote: > You say this will scale better but I am not even sure about that. The > fact is SR-IOV could scale to 256 VFs, but for networking I kind of > doubt the limitation would have been the bus number and would more > likely be issues wit

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Alexander Duyck
On Wed, Dec 16, 2020 at 5:33 AM Jason Gunthorpe wrote: > > On Tue, Dec 15, 2020 at 08:13:21PM -0800, Alexander Duyck wrote: > > > > > Ugh, don't get me started on switchdev. The biggest issue as I see it > > > > with switchev is that you have to have a true switch in order to > > > > really be abl

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-16 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 08:13:21PM -0800, Alexander Duyck wrote: > > > Ugh, don't get me started on switchdev. The biggest issue as I see it > > > with switchev is that you have to have a true switch in order to > > > really be able to use it. > > > > That cuts both ways, suggesting HW with a true

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Leon Romanovsky
On Tue, Dec 15, 2020 at 01:28:05PM -0800, Jakub Kicinski wrote: > On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote: > > > I think the big thing we really should do if we are going to go this > > > route is to look at standardizing what the flavours are that get > > > created by the parent ne

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Parav Pandit
> From: Alexander Duyck > Sent: Wednesday, December 16, 2020 9:43 AM > > > > > That is goal here. This is not about creating just a netdev, this is > > about the whole kit: rdma, netdev, vdpa virtio-net, virtio-mdev. > > One issue is right now we are only seeing the rdma and netdev. It is kind

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Tue, Dec 15, 2020 at 7:04 PM Jason Gunthorpe wrote: > > On Tue, Dec 15, 2020 at 06:19:18PM -0800, Alexander Duyck wrote: > > > > > I would really like to see is a solid standardization of what this is. > > > > Otherwise the comparison is going to be made. Especially since a year > > > > ago Mel

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Tue, Dec 15, 2020 at 5:13 PM Edwin Peer wrote: > > On Tue, Dec 15, 2020 at 10:49 AM Alexander Duyck > wrote: > > > It isn't "SR-IOV done right" it seems more like "VMDq done better". > > I don't think I agree with that assertion. The fact that VMDq can talk > to a common driver still makes VMD

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 06:19:18PM -0800, Alexander Duyck wrote: > > > I would really like to see is a solid standardization of what this is. > > > Otherwise the comparison is going to be made. Especially since a year > > > ago Mellanox was pushing this as an mdev type interface. > > > > mdev was

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 05:12:33PM -0800, Edwin Peer wrote: > 1) More than 256 SFs are possible: Maybe it's about time PCI-SIG > addresses this limit for VFs? They can't, the Bus/Device/Function is limited by protocol and changing that would upend the entire PCI world. Instead PCI-SIG said PASI

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Tue, Dec 15, 2020 at 4:20 PM Jason Gunthorpe wrote: > > On Tue, Dec 15, 2020 at 01:41:04PM -0800, Alexander Duyck wrote: > > > > not just devlink and switchdev, auxbus was also introduced to > > > standardize some of the interfaces. > > > > The auxbus is just there to make up for the fact that

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Edwin Peer
On Tue, Dec 15, 2020 at 10:49 AM Alexander Duyck wrote: > It isn't "SR-IOV done right" it seems more like "VMDq done better". I don't think I agree with that assertion. The fact that VMDq can talk to a common driver still makes VMDq preferable in some respects. Thus, subfunctions do appear to be

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 01:41:04PM -0800, Alexander Duyck wrote: > > not just devlink and switchdev, auxbus was also introduced to > > standardize some of the interfaces. > > The auxbus is just there to make up for the fact that there isn't > another bus type for this though. I would imagine othe

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Tue, Dec 15, 2020 at 12:35 PM Saeed Mahameed wrote: > > On Tue, 2020-12-15 at 11:12 -0800, Alexander Duyck wrote: > > On Mon, Dec 14, 2020 at 10:15 PM Saeed Mahameed > > wrote: > > > On Mon, 2020-12-14 at 17:53 -0800, Alexander Duyck wrote: > > > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahamee

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jakub Kicinski
On Tue, 15 Dec 2020 12:35:20 -0800 Saeed Mahameed wrote: > > I think the big thing we really should do if we are going to go this > > route is to look at standardizing what the flavours are that get > > created by the parent netdevice. Otherwise we are just creating the > > same mess we had with SR

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Jason Gunthorpe
On Tue, Dec 15, 2020 at 10:47:36AM -0800, Alexander Duyck wrote: > > Jason and Saeed explained this in great detail few weeks back in v0 version > > of the patchset at [1], [2] and [3]. > > I better not repeat all of it here again. Please go through it. > > If you may want to read precursor to it

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread David Ahern
On 12/14/20 10:48 PM, Parav Pandit wrote: > >> From: Alexander Duyck >> Sent: Tuesday, December 15, 2020 7:24 AM >> >> On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed >> wrote: >>> >>> Hi Dave, Jakub, Jason, >>> >> >> Just to clarify a few things for myself. You mention virtualization and >> SR-

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Saeed Mahameed
On Tue, 2020-12-15 at 11:12 -0800, Alexander Duyck wrote: > On Mon, Dec 14, 2020 at 10:15 PM Saeed Mahameed > wrote: > > On Mon, 2020-12-14 at 17:53 -0800, Alexander Duyck wrote: > > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > > > wrote: > > > > Hi Dave, Jakub, Jason, > > > > > > > > This

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Saeed Mahameed
On Tue, 2020-12-15 at 10:47 -0800, Alexander Duyck wrote: > On Mon, Dec 14, 2020 at 9:48 PM Parav Pandit > wrote: > > > > > From: Alexander Duyck > > > Sent: Tuesday, December 15, 2020 7:24 AM > > > > > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > > > wrote: > > > > Hi Dave, Jakub, Jason

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Mon, Dec 14, 2020 at 10:15 PM Saeed Mahameed wrote: > > On Mon, 2020-12-14 at 17:53 -0800, Alexander Duyck wrote: > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > > wrote: > > > Hi Dave, Jakub, Jason, > > > > > > This series form Parav was the theme of this mlx5 release cycle, > > > we've

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Mon, Dec 14, 2020 at 9:48 PM Parav Pandit wrote: > > > > From: Alexander Duyck > > Sent: Tuesday, December 15, 2020 7:24 AM > > > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > > wrote: > > > > > > Hi Dave, Jakub, Jason, > > > > > > > Just to clarify a few things for myself. You mention v

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Parav Pandit
> From: Alexander Duyck > Sent: Tuesday, December 15, 2020 9:47 PM > > On Mon, Dec 14, 2020 at 6:44 PM David Ahern wrote: > > > > On 12/14/20 6:53 PM, Alexander Duyck wrote: > > >> example subfunction usage sequence: > > >> --- > > >> Change device to switchdev

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-15 Thread Alexander Duyck
On Mon, Dec 14, 2020 at 6:44 PM David Ahern wrote: > > On 12/14/20 6:53 PM, Alexander Duyck wrote: > >> example subfunction usage sequence: > >> --- > >> Change device to switchdev mode: > >> $ devlink dev eswitch set pci/:06:00.0 mode switchdev > >> > >> Add a

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread Saeed Mahameed
On Mon, 2020-12-14 at 17:53 -0800, Alexander Duyck wrote: > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > wrote: > > Hi Dave, Jakub, Jason, > > > > This series form Parav was the theme of this mlx5 release cycle, > > we've been waiting anxiously for the auxbus infrastructure to make > > it int

RE: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread Parav Pandit
> From: Alexander Duyck > Sent: Tuesday, December 15, 2020 7:24 AM > > On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed > wrote: > > > > Hi Dave, Jakub, Jason, > > > > Just to clarify a few things for myself. You mention virtualization and SR-IOV > in your patch description but you cannot suppor

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread David Ahern
On 12/14/20 6:53 PM, Alexander Duyck wrote: >> example subfunction usage sequence: >> --- >> Change device to switchdev mode: >> $ devlink dev eswitch set pci/:06:00.0 mode switchdev >> >> Add a devlink port of subfunction flaovur: >> $ devlink port add pci/:

Re: [net-next v4 00/15] Add mlx5 subfunction support

2020-12-14 Thread Alexander Duyck
On Mon, Dec 14, 2020 at 1:49 PM Saeed Mahameed wrote: > > Hi Dave, Jakub, Jason, > > This series form Parav was the theme of this mlx5 release cycle, > we've been waiting anxiously for the auxbus infrastructure to make it into > the kernel, and now as the auxbus is in and all the stars are aligned