Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-06-08 Thread Jason Gunthorpe
On Tue, Jun 08, 2021 at 10:44:31AM +1000, David Gibson wrote: > When you say "not using a drivers/iommu IOMMU interface" do you > basically mean the device doesn't do DMA? No, I mean the device doesn't use iommu_map() to manage the DMA mappings. vfio_iommu_type1 has a special code path that mde

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-06-07 Thread David Gibson
On Tue, Jun 01, 2021 at 09:57:12AM -0300, Jason Gunthorpe wrote: > On Tue, Jun 01, 2021 at 02:03:33PM +1000, David Gibson wrote: > > On Thu, May 27, 2021 at 03:48:47PM -0300, Jason Gunthorpe wrote: > > > On Thu, May 27, 2021 at 02:58:30PM +1000, David Gibson wrote: > > > > On Tue, May 25, 2021 at 0

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-06-01 Thread Jason Gunthorpe
On Tue, Jun 01, 2021 at 02:03:33PM +1000, David Gibson wrote: > On Thu, May 27, 2021 at 03:48:47PM -0300, Jason Gunthorpe wrote: > > On Thu, May 27, 2021 at 02:58:30PM +1000, David Gibson wrote: > > > On Tue, May 25, 2021 at 04:52:57PM -0300, Jason Gunthorpe wrote: > > > > On Wed, May 26, 2021 at 1

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-06-01 Thread David Gibson
On Thu, May 27, 2021 at 11:55:00PM +0530, Kirti Wankhede wrote: > > > On 5/27/2021 10:30 AM, David Gibson wrote: > > On Wed, May 26, 2021 at 02:48:03AM +0530, Kirti Wankhede wrote: > > > > > > > > > On 5/26/2021 1:22 AM, Jason Gunthorpe wrote: > > > > On Wed, May 26, 2021 at 12:56:30AM +0530, K

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-06-01 Thread David Gibson
On Thu, May 27, 2021 at 04:06:20PM -0300, Jason Gunthorpe wrote: > On Thu, May 27, 2021 at 02:53:42PM +1000, David Gibson wrote: > > > > > If the physical device had a bug which meant the mdevs *weren't* > > > > properly isolated from each other, then those mdevs would share a > > > > group, and y

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-06-01 Thread David Gibson
On Thu, May 27, 2021 at 03:48:47PM -0300, Jason Gunthorpe wrote: > On Thu, May 27, 2021 at 02:58:30PM +1000, David Gibson wrote: > > On Tue, May 25, 2021 at 04:52:57PM -0300, Jason Gunthorpe wrote: > > > On Wed, May 26, 2021 at 12:56:30AM +0530, Kirti Wankhede wrote: > > > > > > > 2. iommu backed

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-27 Thread Jason Gunthorpe
On Thu, May 27, 2021 at 02:53:42PM +1000, David Gibson wrote: > > > If the physical device had a bug which meant the mdevs *weren't* > > > properly isolated from each other, then those mdevs would share a > > > group, and you *would* care about it. Depending on how the isolation > > > failed the

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-27 Thread Jason Gunthorpe
On Thu, May 27, 2021 at 02:58:30PM +1000, David Gibson wrote: > On Tue, May 25, 2021 at 04:52:57PM -0300, Jason Gunthorpe wrote: > > On Wed, May 26, 2021 at 12:56:30AM +0530, Kirti Wankhede wrote: > > > > > 2. iommu backed mdev devices for SRIOV where mdev device is created per > > > VF (mdev devi

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-27 Thread Kirti Wankhede
On 5/27/2021 10:30 AM, David Gibson wrote: On Wed, May 26, 2021 at 02:48:03AM +0530, Kirti Wankhede wrote: On 5/26/2021 1:22 AM, Jason Gunthorpe wrote: On Wed, May 26, 2021 at 12:56:30AM +0530, Kirti Wankhede wrote: 2. iommu backed mdev devices for SRIOV where mdev device is created per

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-27 Thread David Gibson
On Mon, May 24, 2021 at 08:37:44PM -0300, Jason Gunthorpe wrote: > On Mon, May 24, 2021 at 05:52:58PM +1000, David Gibson wrote: > > > > > I don't really see a semantic distinction between "always one-device > > > > groups" and "groups don't matter". Really the only way you can afford > > > > to

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-27 Thread David Gibson
On Tue, May 25, 2021 at 04:52:57PM -0300, Jason Gunthorpe wrote: > On Wed, May 26, 2021 at 12:56:30AM +0530, Kirti Wankhede wrote: > > > 2. iommu backed mdev devices for SRIOV where mdev device is created per > > VF (mdev device == VF device) then that mdev device has same iommu > > protection sco

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-27 Thread David Gibson
On Wed, May 26, 2021 at 02:48:03AM +0530, Kirti Wankhede wrote: > > > On 5/26/2021 1:22 AM, Jason Gunthorpe wrote: > > On Wed, May 26, 2021 at 12:56:30AM +0530, Kirti Wankhede wrote: > > > > > 2. iommu backed mdev devices for SRIOV where mdev device is created per > > > VF (mdev device == VF dev

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-26 Thread Jason Gunthorpe
On Wed, May 26, 2021 at 12:59:05PM -0600, Alex Williamson wrote: > A driver provided sysfs attribute would obviously fill the short > term gap, long term maybe this would be standardized via netlink. It > seems a bit analogous to setting the MAC address for a VF on an SR-IOV > NIC or VF namespace

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-26 Thread Alex Williamson
On Wed, 26 May 2021 23:40:02 +0530 Kirti Wankhede wrote: > On 5/26/2021 4:22 AM, Alex Williamson wrote: > > On Wed, 26 May 2021 00:56:30 +0530 > > Kirti Wankhede wrote: > > > >> On 5/25/2021 5:07 AM, Jason Gunthorpe wrote: > >>> On Mon, May 24, 2021 at 05:52:58PM +1000, David Gibson wrote:

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-26 Thread Kirti Wankhede
On 5/26/2021 4:22 AM, Alex Williamson wrote: On Wed, 26 May 2021 00:56:30 +0530 Kirti Wankhede wrote: On 5/25/2021 5:07 AM, Jason Gunthorpe wrote: On Mon, May 24, 2021 at 05:52:58PM +1000, David Gibson wrote: I don't really see a semantic distinction between "always one-device groups"

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-25 Thread Alex Williamson
On Wed, 26 May 2021 00:56:30 +0530 Kirti Wankhede wrote: > On 5/25/2021 5:07 AM, Jason Gunthorpe wrote: > > On Mon, May 24, 2021 at 05:52:58PM +1000, David Gibson wrote: > > > I don't really see a semantic distinction between "always one-device > groups" and "groups don't matter". R

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-25 Thread Kirti Wankhede
On 5/26/2021 1:22 AM, Jason Gunthorpe wrote: On Wed, May 26, 2021 at 12:56:30AM +0530, Kirti Wankhede wrote: 2. iommu backed mdev devices for SRIOV where mdev device is created per VF (mdev device == VF device) then that mdev device has same iommu protection scope as VF associated to it. T

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-25 Thread Jason Gunthorpe
On Wed, May 26, 2021 at 12:56:30AM +0530, Kirti Wankhede wrote: > 2. iommu backed mdev devices for SRIOV where mdev device is created per > VF (mdev device == VF device) then that mdev device has same iommu > protection scope as VF associated to it. This doesn't require, and certainly shouldn't

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-25 Thread Kirti Wankhede
On 5/25/2021 5:07 AM, Jason Gunthorpe wrote: On Mon, May 24, 2021 at 05:52:58PM +1000, David Gibson wrote: I don't really see a semantic distinction between "always one-device groups" and "groups don't matter". Really the only way you can afford to not care about groups is if they're single

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-24 Thread Jason Gunthorpe
On Mon, May 24, 2021 at 05:52:58PM +1000, David Gibson wrote: > > > I don't really see a semantic distinction between "always one-device > > > groups" and "groups don't matter". Really the only way you can afford > > > to not care about groups is if they're singletons. > > > > The kernel driver

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-24 Thread David Gibson
On Thu, May 13, 2021 at 10:59:38AM -0300, Jason Gunthorpe wrote: > On Thu, May 13, 2021 at 03:48:19PM +1000, David Gibson wrote: > > On Mon, May 03, 2021 at 01:15:18PM -0300, Jason Gunthorpe wrote: > > > On Thu, Apr 29, 2021 at 01:04:05PM +1000, David Gibson wrote: > > > > Again, I don't know enoug

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-24 Thread David Gibson
On Thu, May 13, 2021 at 10:50:30AM -0300, Jason Gunthorpe wrote: > On Thu, May 13, 2021 at 04:07:07PM +1000, David Gibson wrote: > > On Wed, May 05, 2021 at 01:39:02PM -0300, Jason Gunthorpe wrote: > > > On Wed, May 05, 2021 at 02:28:53PM +1000, Alexey Kardashevskiy wrote: > > > > > > > This is a

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-13 Thread Jason Gunthorpe
On Thu, May 13, 2021 at 03:48:19PM +1000, David Gibson wrote: > On Mon, May 03, 2021 at 01:15:18PM -0300, Jason Gunthorpe wrote: > > On Thu, Apr 29, 2021 at 01:04:05PM +1000, David Gibson wrote: > > > Again, I don't know enough about VDPA to make sense of that. Are we > > > essentially talking non

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-13 Thread Jason Gunthorpe
On Thu, May 13, 2021 at 04:07:07PM +1000, David Gibson wrote: > On Wed, May 05, 2021 at 01:39:02PM -0300, Jason Gunthorpe wrote: > > On Wed, May 05, 2021 at 02:28:53PM +1000, Alexey Kardashevskiy wrote: > > > > > This is a good feature in general when let's say there is a linux > > > supported >

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-13 Thread Jason Gunthorpe
On Thu, May 13, 2021 at 04:01:20PM +1000, David Gibson wrote: > But.. even if you're exposing page tables to userspace.. with hardware > that has explicit support for nesting you can probably expose the hw > tables directly which is great for the cases that works for. But > surely for older IOMMU

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-12 Thread Tian, Kevin
> From: David Gibson > Sent: Thursday, May 13, 2021 2:01 PM > > > > But this definitely all becomes HW specific. > > > > For instance I want to have an ARM vIOMMU driver it needs to do some > > > > ret = ioctl(ioasid_fd, CREATE_NESTED_IOASID, [page table format is > ARMvXXX]) > > if (ret == -EOP

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-12 Thread David Gibson
On Wed, May 05, 2021 at 01:39:02PM -0300, Jason Gunthorpe wrote: > On Wed, May 05, 2021 at 02:28:53PM +1000, Alexey Kardashevskiy wrote: > > > This is a good feature in general when let's say there is a linux supported > > device which has a proprietary device firmware update tool which only exist

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-12 Thread David Gibson
On Mon, May 03, 2021 at 01:15:18PM -0300, Jason Gunthorpe wrote: > On Thu, Apr 29, 2021 at 01:04:05PM +1000, David Gibson wrote: > > Again, I don't know enough about VDPA to make sense of that. Are we > > essentially talking non-PCI virtual devices here? In which case you > > could define the VDP

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-12 Thread David Gibson
On Tue, May 04, 2021 at 03:15:37PM -0300, Jason Gunthorpe wrote: > On Tue, May 04, 2021 at 01:54:55PM +1000, David Gibson wrote: > > On Mon, May 03, 2021 at 01:05:30PM -0300, Jason Gunthorpe wrote: > > > On Thu, Apr 29, 2021 at 01:20:22PM +1000, David Gibson wrote: > > > > > There is a certain appe

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Wednesday, May 12, 2021 8:25 AM > > On Wed, May 12, 2021 at 12:21:24AM +, Tian, Kevin wrote: > > > > Basically each RID knows based on its kernel drivers if it is a local > > > or global RID and the ioasid knob can further fine tune this for any > > > other sp

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Jason Gunthorpe
On Wed, May 12, 2021 at 12:21:24AM +, Tian, Kevin wrote: > > Basically each RID knows based on its kernel drivers if it is a local > > or global RID and the ioasid knob can further fine tune this for any > > other specialty cases. > > It's fine if you insist on this way. Then we leave it to u

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Wednesday, May 12, 2021 7:40 AM > > On Tue, May 11, 2021 at 10:51:40PM +, Tian, Kevin wrote: > > > From: Jason Gunthorpe > > > Sent: Tuesday, May 11, 2021 10:39 PM > > > > > > On Tue, May 11, 2021 at 09:10:03AM +, Tian, Kevin wrote: > > > > > > > 3) SRIOV,

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Jason Gunthorpe
On Tue, May 11, 2021 at 10:51:40PM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Tuesday, May 11, 2021 10:39 PM > > > > On Tue, May 11, 2021 at 09:10:03AM +, Tian, Kevin wrote: > > > > > 3) SRIOV, ENQCMD (Intel): > > > - "PASID global" with host-allocated PASIDs; > > > -

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Tian, Kevin
> From: Liu Yi L > Sent: Tuesday, May 11, 2021 9:25 PM > > On Tue, 11 May 2021 09:10:03 +, Tian, Kevin wrote: > > > > From: Jason Gunthorpe > > > Sent: Monday, May 10, 2021 8:37 PM > > > > > [...] > > > > gPASID!=hPASID has a problem when assigning a physical device which > > > > supports bo

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Tuesday, May 11, 2021 10:39 PM > > On Tue, May 11, 2021 at 09:10:03AM +, Tian, Kevin wrote: > > > 3) SRIOV, ENQCMD (Intel): > > - "PASID global" with host-allocated PASIDs; > > - PASID table managed by host (in HPA space); > > - all RIDs bound to t

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Jason Gunthorpe
On Tue, May 11, 2021 at 09:10:03AM +, Tian, Kevin wrote: > 3) SRIOV, ENQCMD (Intel): > - "PASID global" with host-allocated PASIDs; > - PASID table managed by host (in HPA space); > - all RIDs bound to this ioasid_fd use the global pool; > - however, exposing global PAS

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Liu Yi L
On Tue, 11 May 2021 09:10:03 +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Monday, May 10, 2021 8:37 PM > > > [...] > > > gPASID!=hPASID has a problem when assigning a physical device which > > > supports both shared work queue (ENQCMD with PASID in MSR) > > > and dedicated wor

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-11 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Monday, May 10, 2021 8:37 PM > [...] > > gPASID!=hPASID has a problem when assigning a physical device which > > supports both shared work queue (ENQCMD with PASID in MSR) > > and dedicated work queue (PASID in device register) to a guest > > process which is assoc

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-10 Thread Jacob Pan
Hi Jason, On Mon, 10 May 2021 20:45:00 -0300, Jason Gunthorpe wrote: > On Mon, May 10, 2021 at 03:28:54PM -0700, Jacob Pan wrote: > > > To satisfy your "give me a PASID for this RID" proposal, can we just use > > the RID's struct device as the token? Also add a type field to > > explicitly indi

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-10 Thread Jason Gunthorpe
On Mon, May 10, 2021 at 03:28:54PM -0700, Jacob Pan wrote: > To satisfy your "give me a PASID for this RID" proposal, can we just use > the RID's struct device as the token? Also add a type field to explicitly > indicate global vs per-set(per-RID). i.e. You've got it backwards, the main behavior

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-10 Thread Jacob Pan
Hi Jason, On Mon, 10 May 2021 13:39:56 -0300, Jason Gunthorpe wrote: > I still think it is smarter to push a group of RID's into a global > allocation group and accept there are potential downsides with that > than to try to force a global allocation group on every RID and then > try to fix the

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-10 Thread Jason Gunthorpe
On Mon, May 10, 2021 at 09:22:12AM -0700, Raj, Ashok wrote: > Those vIOMMU's will have a capability that it supports PASID allocation > interface. When you allocate you can say what type of PASID you need > (shared vs local) and Qemu will obtain either within the local range, or in > the shared ra

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-10 Thread Raj, Ashok
On Mon, May 10, 2021 at 12:31:11PM -0300, Jason Gunthorpe wrote: > On Mon, May 10, 2021 at 08:25:02AM -0700, Raj, Ashok wrote: > > > Global PASID doesn't break anything, but giving that control to vIOMMU > > doesn't seem right. When we have mixed uses cases like hardware that > > supports shared w

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-10 Thread Jason Gunthorpe
On Mon, May 10, 2021 at 08:25:02AM -0700, Raj, Ashok wrote: > Global PASID doesn't break anything, but giving that control to vIOMMU > doesn't seem right. When we have mixed uses cases like hardware that > supports shared wq and SRIOV devices that need PASIDs we need to > comprehend how they will

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-10 Thread Raj, Ashok
On Mon, May 10, 2021 at 09:37:29AM -0300, Jason Gunthorpe wrote: > On Sat, May 08, 2021 at 09:56:59AM +, Tian, Kevin wrote: > > > From: Raj, Ashok > > > Sent: Friday, May 7, 2021 12:33 AM > > > > > > > Basically it means when the guest's top level IOASID is created for > > > > nesting that IO

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-10 Thread Jason Gunthorpe
On Sat, May 08, 2021 at 09:56:59AM +, Tian, Kevin wrote: > > From: Raj, Ashok > > Sent: Friday, May 7, 2021 12:33 AM > > > > > Basically it means when the guest's top level IOASID is created for > > > nesting that IOASID claims all PASID's on the RID and excludes any > > > PASID IOASIDs from

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-09 Thread Lu Baolu
On 5/8/21 3:31 PM, Tian, Kevin wrote: From: Alex Williamson Sent: Saturday, May 8, 2021 1:06 AM Those are the main ones I can think of. It is nice to have a simple map/unmap interface, I'd hope that a new /dev/ioasid interface wouldn't raise the barrier to entry too high, but the user needs to

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-08 Thread Tian, Kevin
> From: Raj, Ashok > Sent: Friday, May 7, 2021 12:33 AM > > > Basically it means when the guest's top level IOASID is created for > > nesting that IOASID claims all PASID's on the RID and excludes any > > PASID IOASIDs from existing on the RID now or in future. > > The way to look at it this is

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-08 Thread Tian, Kevin
> From: Alex Williamson > Sent: Saturday, May 8, 2021 1:06 AM > > > > Those are the main ones I can think of. It is nice to have a simple > > > map/unmap interface, I'd hope that a new /dev/ioasid interface wouldn't > > > raise the barrier to entry too high, but the user needs to have the > > >

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-08 Thread Liu Yi L
Hi Jason, On Wed, 5 May 2021 19:21:20 -0300, Jason Gunthorpe wrote: > On Wed, May 05, 2021 at 01:04:46PM -0700, Jacob Pan wrote: > > Hi Jason, > > > > On Wed, 5 May 2021 15:00:23 -0300, Jason Gunthorpe wrote: > > > > > On Wed, May 05, 2021 at 10:22:59AM -0700, Jacob Pan wrote: > > > > > >

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Saturday, May 8, 2021 1:11 AM > > On Fri, May 07, 2021 at 11:06:14AM -0600, Alex Williamson wrote: > > > We had tossed around an idea of a super-container with vfio, it's maybe > > something we'd want to incorporate into this design. For instance, if > > memory co

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Thursday, April 29, 2021 4:46 AM > > > > I think the name IOASID is fine for the uAPI, the kernel version can > > > be called ioasid_id or something. > > > > ioasid is already an id and then ioasid_id just adds confusion. Another > > point is that ioasid is current

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Jacob Pan
Hi Jason, On Fri, 7 May 2021 16:28:10 -0300, Jason Gunthorpe wrote: > > The unanswered question is how do we plumb from vIOMMU without a custom > > allocator to get a system wide PASID? > > PASID allocation is part of the iommu driver, it really shouldn't be > global. > In the current code,

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Jason Gunthorpe
On Fri, May 07, 2021 at 12:23:25PM -0700, Raj, Ashok wrote: > Hi Jason > > - Removed lizefan's email due to bounces... > > On Fri, May 07, 2021 at 03:20:50PM -0300, Jason Gunthorpe wrote: > > On Fri, May 07, 2021 at 11:14:58AM -0700, Raj, Ashok wrote: > > > On Fri, May 07, 2021 at 02:20:51PM -03

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Raj, Ashok
Hi Jason - Removed lizefan's email due to bounces... On Fri, May 07, 2021 at 03:20:50PM -0300, Jason Gunthorpe wrote: > On Fri, May 07, 2021 at 11:14:58AM -0700, Raj, Ashok wrote: > > On Fri, May 07, 2021 at 02:20:51PM -0300, Jason Gunthorpe wrote: > > > On Thu, May 06, 2021 at 09:32:40AM -0700,

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Jason Gunthorpe
On Fri, May 07, 2021 at 11:14:58AM -0700, Raj, Ashok wrote: > On Fri, May 07, 2021 at 02:20:51PM -0300, Jason Gunthorpe wrote: > > On Thu, May 06, 2021 at 09:32:40AM -0700, Raj, Ashok wrote: > > > > > For platforms that support ENQCMD, it is required to mandate PASIDs are > > > global across the e

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Raj, Ashok
On Fri, May 07, 2021 at 02:20:51PM -0300, Jason Gunthorpe wrote: > On Thu, May 06, 2021 at 09:32:40AM -0700, Raj, Ashok wrote: > > > For platforms that support ENQCMD, it is required to mandate PASIDs are > > global across the entire system. Maybe its better to call them gPASID for > > guest and h

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Jason Gunthorpe
On Thu, May 06, 2021 at 09:32:40AM -0700, Raj, Ashok wrote: > For platforms that support ENQCMD, it is required to mandate PASIDs are > global across the entire system. Maybe its better to call them gPASID for > guest and hPASID for host. Short reason being gPASID->hPASID is a guest > wide mapping

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Jason Gunthorpe
On Fri, May 07, 2021 at 11:06:14AM -0600, Alex Williamson wrote: > We had tossed around an idea of a super-container with vfio, it's maybe > something we'd want to incorporate into this design. For instance, if > memory could be pre-registered with a super container, which would > handle the lock

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Alex Williamson
On Fri, 7 May 2021 07:36:49 + "Tian, Kevin" wrote: > > From: Alex Williamson > > Sent: Wednesday, April 28, 2021 11:06 PM > > > > On Wed, 28 Apr 2021 06:34:11 + > > "Tian, Kevin" wrote: > > > > > > Can you or Alex elaborate where the complexity and performance problem > > > locate in V

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Jason Gunthorpe
On Fri, May 07, 2021 at 07:36:49AM +, Tian, Kevin wrote: > for /dev/ioasid there is still an open whether an process is allowed to > open /dev/ioasid once or multiple times. If there is only one ioasid_fd > per process, the accounting can be made accurately. otherwise the > same problem still

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Wednesday, May 5, 2021 1:13 AM > > On Wed, Apr 28, 2021 at 06:58:19AM +, Tian, Kevin wrote: > > > From: Jason Gunthorpe > > > Sent: Wednesday, April 28, 2021 1:12 AM > > > > > [...] > > > > As Alex says, if this line fails because of the group restrictions, >

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-07 Thread Tian, Kevin
> From: Alex Williamson > Sent: Wednesday, April 28, 2021 11:06 PM > > On Wed, 28 Apr 2021 06:34:11 + > "Tian, Kevin" wrote: > > > > From: Jason Gunthorpe > > > Sent: Monday, April 26, 2021 8:38 PM > > > > > [...] > > > > Want to hear your opinion for one open here. There is no doubt that

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-06 Thread Raj, Ashok
Hi Jason On Thu, May 06, 2021 at 09:27:30AM -0300, Jason Gunthorpe wrote: > On Thu, May 06, 2021 at 09:23:48AM +0200, Jean-Philippe Brucker wrote: > > On Wed, May 05, 2021 at 01:04:46PM -0700, Jacob Pan wrote: > > > > > For ARM, since the guest owns the per device PASID table. There is no > > > >

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-06 Thread Jason Gunthorpe
On Thu, May 06, 2021 at 09:23:48AM +0200, Jean-Philippe Brucker wrote: > On Wed, May 05, 2021 at 01:04:46PM -0700, Jacob Pan wrote: > > > > For ARM, since the guest owns the per device PASID table. There is no > > > > need to allocate PASIDs from the host nor the hypervisor. Without SWQ, > > > > th

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-06 Thread Jason Gunthorpe
On Wed, May 05, 2021 at 04:23:19PM -0700, Raj, Ashok wrote: > > Which implies the API to the iommu driver should be more like: > > > > 'assign an IOASID to this RID and return the PASID' > > 'reserve a PASID from every RID' > > I don't think this has any decent change of success. Its rather r

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-06 Thread Jean-Philippe Brucker
On Wed, May 05, 2021 at 01:04:46PM -0700, Jacob Pan wrote: > > > For ARM, since the guest owns the per device PASID table. There is no > > > need to allocate PASIDs from the host nor the hypervisor. Without SWQ, > > > there is no need for global PASID/SSID either. So PASID being global > > > for AR

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-05 Thread Raj, Ashok
On Wed, May 05, 2021 at 07:21:20PM -0300, Jason Gunthorpe wrote: > On Wed, May 05, 2021 at 01:04:46PM -0700, Jacob Pan wrote: > > Hi Jason, > > > > On Wed, 5 May 2021 15:00:23 -0300, Jason Gunthorpe wrote: > > > > > On Wed, May 05, 2021 at 10:22:59AM -0700, Jacob Pan wrote: > > > > > > > Global

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-05 Thread Jason Gunthorpe
On Wed, May 05, 2021 at 01:04:46PM -0700, Jacob Pan wrote: > Hi Jason, > > On Wed, 5 May 2021 15:00:23 -0300, Jason Gunthorpe wrote: > > > On Wed, May 05, 2021 at 10:22:59AM -0700, Jacob Pan wrote: > > > > > Global and pluggable are for slightly separate reasons. > > > - We need global PASID on

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-05 Thread Jacob Pan
Hi Jason, On Wed, 5 May 2021 15:00:23 -0300, Jason Gunthorpe wrote: > On Wed, May 05, 2021 at 10:22:59AM -0700, Jacob Pan wrote: > > > Global and pluggable are for slightly separate reasons. > > - We need global PASID on VT-d in that we need to support shared > > workqueues (SWQ). E.g. One SWQ

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-05 Thread Jason Gunthorpe
On Wed, May 05, 2021 at 10:22:59AM -0700, Jacob Pan wrote: > Global and pluggable are for slightly separate reasons. > - We need global PASID on VT-d in that we need to support shared > workqueues (SWQ). E.g. One SWQ can be wrapped into two mdevs then assigned > to two VMs. Each VM uses its privat

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-05 Thread Jacob Pan
Hi Jason, On Tue, 4 May 2021 20:15:30 -0300, Jason Gunthorpe wrote: > On Tue, May 04, 2021 at 03:11:54PM -0700, Jacob Pan wrote: > > > > It is a weird way to use xarray to have a structure which > > > itself is just a wrapper around another RCU protected structure. > > > > > > Make the caller

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-05 Thread Jason Gunthorpe
On Wed, May 05, 2021 at 02:28:53PM +1000, Alexey Kardashevskiy wrote: > This is a good feature in general when let's say there is a linux supported > device which has a proprietary device firmware update tool which only exists > as an x86 binary and your hardware is not x86 - running qemu + vfio i

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-05 Thread Auger Eric
Hi Jason, On 4/29/21 10:04 PM, Jason Gunthorpe wrote: > On Thu, Apr 29, 2021 at 03:26:55PM +0200, Auger Eric wrote: >> From the pseudo code, >> >> gpa_ioasid_id = ioctl(ioasid_fd, CREATE_IOASID, ..) >> ioctl(ioasid_fd, SET_IOASID_PAGE_TABLES, ..) >> >> I fail to understand whether the SET_IOAS

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Alexey Kardashevskiy
On 05/05/2021 04:15, Jason Gunthorpe wrote: On Tue, May 04, 2021 at 01:54:55PM +1000, David Gibson wrote: On Mon, May 03, 2021 at 01:05:30PM -0300, Jason Gunthorpe wrote: On Thu, Apr 29, 2021 at 01:20:22PM +1000, David Gibson wrote: There is a certain appeal to having some 'PPC_TCE_CREATE_S

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Jason Gunthorpe
On Tue, May 04, 2021 at 03:11:54PM -0700, Jacob Pan wrote: > > It is a weird way to use xarray to have a structure which > > itself is just a wrapper around another RCU protected structure. > > > > Make the caller supply the ioasid_data memory, embedded in its own > > element, get rid of the void

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Jacob Pan
Hi Jason, On Tue, 4 May 2021 15:00:50 -0300, Jason Gunthorpe wrote: > On Tue, May 04, 2021 at 08:41:48AM -0700, Jacob Pan wrote: > > > > > > > > (also looking at ioasid.c, why do we need such a thin and odd > > > > wrapper around xarray?) > > > > > > > > > > I'll leave it to Jean and Jaco

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Jason Gunthorpe
On Tue, May 04, 2021 at 01:54:55PM +1000, David Gibson wrote: > On Mon, May 03, 2021 at 01:05:30PM -0300, Jason Gunthorpe wrote: > > On Thu, Apr 29, 2021 at 01:20:22PM +1000, David Gibson wrote: > > > > There is a certain appeal to having some > > > > 'PPC_TCE_CREATE_SPECIAL_IOASID' entry point tha

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Jason Gunthorpe
On Tue, May 04, 2021 at 08:41:48AM -0700, Jacob Pan wrote: > > > > > > (also looking at ioasid.c, why do we need such a thin and odd wrapper > > > around xarray?) > > > > > > > I'll leave it to Jean and Jacob. > Could you elaborate? I mean stuff like this: int ioasid_set_data(ioasid_t ioasi

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Jason Gunthorpe
On Wed, Apr 28, 2021 at 06:58:19AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, April 28, 2021 1:12 AM > > > [...] > > > As Alex says, if this line fails because of the group restrictions, > > > that's not great because it's not very obvious what's gone wrong. > > >

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Jason Gunthorpe
On Tue, May 04, 2021 at 09:22:55AM -0700, Jacob Pan wrote: > Hi Jason, > > On Wed, 28 Apr 2021 17:46:06 -0300, Jason Gunthorpe wrote: > > > > > I think the name IOASID is fine for the uAPI, the kernel version can > > > > be called ioasid_id or something. > > > > > > ioasid is already an id an

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Jacob Pan
Hi Jason, On Wed, 28 Apr 2021 17:46:06 -0300, Jason Gunthorpe wrote: > > > I think the name IOASID is fine for the uAPI, the kernel version can > > > be called ioasid_id or something. > > > > ioasid is already an id and then ioasid_id just adds confusion. Another > > point is that ioasid is c

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-04 Thread Jacob Pan
Hi Kevin, On Wed, 28 Apr 2021 06:34:11 +, "Tian, Kevin" wrote: > > > > (also looking at ioasid.c, why do we need such a thin and odd wrapper > > around xarray?) > > > > I'll leave it to Jean and Jacob. I am not sure whether you are referring to the current ioasid.c or the changes propos

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-03 Thread David Gibson
On Mon, May 03, 2021 at 01:05:30PM -0300, Jason Gunthorpe wrote: > On Thu, Apr 29, 2021 at 01:20:22PM +1000, David Gibson wrote: > > > There is a certain appeal to having some > > > 'PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra > > > information like windows that can be optio

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-03 Thread Jason Gunthorpe
On Thu, Apr 29, 2021 at 01:04:05PM +1000, David Gibson wrote: > Again, I don't know enough about VDPA to make sense of that. Are we > essentially talking non-PCI virtual devices here? In which case you > could define the VDPA "bus" to always have one-device groups. It is much worse than that. W

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-05-03 Thread Jason Gunthorpe
On Thu, Apr 29, 2021 at 01:20:22PM +1000, David Gibson wrote: > > There is a certain appeal to having some > > 'PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra > > information like windows that can be optionally called by the viommu > > driver and it remains well defined and des

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-29 Thread Jason Gunthorpe
On Thu, Apr 29, 2021 at 03:26:55PM +0200, Auger Eric wrote: > From the pseudo code, > > gpa_ioasid_id = ioctl(ioasid_fd, CREATE_IOASID, ..) > ioctl(ioasid_fd, SET_IOASID_PAGE_TABLES, ..) > > I fail to understand whether the SET_IOASID_PAGE_TABLES would apply to > the whole IOASIDs within /dev

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-29 Thread Auger Eric
Hi, On 4/22/21 2:10 PM, Jason Gunthorpe wrote: > On Thu, Apr 22, 2021 at 08:34:32AM +, Tian, Kevin wrote: > >> The shim layer could be considered as a new iommu backend in VFIO, >> which connects VFIO iommu ops to the internal helpers in >> drivers/ioasid. > > It may be the best we can do be

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-29 Thread Auger Eric
Hi, On 4/22/21 2:10 PM, Jason Gunthorpe wrote: > On Thu, Apr 22, 2021 at 08:34:32AM +, Tian, Kevin wrote: > >> The shim layer could be considered as a new iommu backend in VFIO, >> which connects VFIO iommu ops to the internal helpers in >> drivers/ioasid. > > It may be the best we can do be

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-29 Thread Auger Eric
Hi, On 4/23/21 1:49 PM, Jason Gunthorpe wrote: > On Fri, Apr 23, 2021 at 09:06:44AM +, Tian, Kevin wrote: > >> Or could we still have just one /dev/ioasid but allow userspace to create >> multiple gpa_ioasid_id's each associated to a different iommu domain? >> Then the compatibility check wi

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-28 Thread David Gibson
On Wed, Apr 28, 2021 at 11:56:22AM -0300, Jason Gunthorpe wrote: > On Wed, Apr 28, 2021 at 10:58:29AM +1000, David Gibson wrote: > > On Tue, Apr 27, 2021 at 02:12:12PM -0300, Jason Gunthorpe wrote: > > > On Tue, Apr 27, 2021 at 03:08:46PM +1000, David Gibson wrote: > > > > > Starting from a BDF the

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-28 Thread David Gibson
On Wed, Apr 28, 2021 at 09:21:49PM -0300, Jason Gunthorpe wrote: > On Wed, Apr 28, 2021 at 11:23:39AM +1000, David Gibson wrote: > > > Yes. My proposed model for a unified interface would be that when you > > create a new container/IOASID, *no* IOVAs are valid. > > Hurm, it is quite tricky. All

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-28 Thread Jason Gunthorpe
On Wed, Apr 28, 2021 at 11:23:39AM +1000, David Gibson wrote: > Yes. My proposed model for a unified interface would be that when you > create a new container/IOASID, *no* IOVAs are valid. Hurm, it is quite tricky. All IOMMUs seem to have a dead zone around the MSI window, so negotiating this al

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-28 Thread Jason Gunthorpe
On Wed, Apr 28, 2021 at 06:34:11AM +, Tian, Kevin wrote: > > If /dev/ioasid is single HW page table only then I would focus on that > > implementation and leave it to userspace to span different > > /dev/ioasids if needed. > > > > > OK, now I see where the disconnection comes from. In my cont

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-28 Thread Jason Gunthorpe
On Wed, Apr 28, 2021 at 07:47:56AM +, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Wednesday, April 28, 2021 1:12 AM > > > [...] > > One option is VFIO can keep its group FD but nothing else will have > > anthing like it. However I don't much like the idea that VFIO will > > have a

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-28 Thread Alex Williamson
On Wed, 28 Apr 2021 06:34:11 + "Tian, Kevin" wrote: > > From: Jason Gunthorpe > > Sent: Monday, April 26, 2021 8:38 PM > > > [...] > > > Want to hear your opinion for one open here. There is no doubt that > > > an ioasid represents a HW page table when the table is constructed by > > > us

Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-28 Thread Jason Gunthorpe
On Wed, Apr 28, 2021 at 10:58:29AM +1000, David Gibson wrote: > On Tue, Apr 27, 2021 at 02:12:12PM -0300, Jason Gunthorpe wrote: > > On Tue, Apr 27, 2021 at 03:08:46PM +1000, David Gibson wrote: > > > > Starting from a BDF the general pseudo code is: > > > > device_name = first_directory_of("/sys/

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-28 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Wednesday, April 28, 2021 1:12 AM > [...] > One option is VFIO can keep its group FD but nothing else will have > anthing like it. However I don't much like the idea that VFIO will > have a special and unique programming model to do that same things > other subsyst

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-27 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Wednesday, April 28, 2021 1:12 AM > [...] > > As Alex says, if this line fails because of the group restrictions, > > that's not great because it's not very obvious what's gone wrong. > > Okay, that is fair, but let's solve that problem directly. For > instance n

RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs

2021-04-27 Thread Tian, Kevin
> From: Jason Gunthorpe > Sent: Monday, April 26, 2021 8:38 PM > [...] > > Want to hear your opinion for one open here. There is no doubt that > > an ioasid represents a HW page table when the table is constructed by > > userspace and then linked to the IOMMU through the bind/unbind > > API. But

  1   2   3   >