Hi Jason,
On Tue, 27 May 2025 21:04:25 -0300
Jason Gunthorpe <j...@ziepe.ca> wrote:
> On Tue, May 27, 2025 at 01:54:28PM -0700, Shyam Saini wrote:
> > > The above is the only place that creates a IOMMU_RESV_SW_MSI so
> > > it is definately called and used, right? If not where does your
> > > IOMMU_RESV_SW_MSI come from?
> >
> > code tracing and printks in that code path suggests
> > iommu_dma_get_resv_regions() called by vfio-pci driver,
>
> Yes, I know it is, that is how it setups the SW_MSI.
>
> > > As above, I've asked a few times now if your resv_regions() is
> > > correct, meaning there is a reserved range covering the address
> > > space that doesn't have working translation. That means
> > > iommu_get_resv_regions() returns such a range.
> >
> > sorry about missing that, i see msi iova being reserved:
> >
> > cat /sys/kernel/iommu_groups/*/reserved_regions
> > 0x0000000008000000 0x00000000080fffff msi
> > 0x0000000008000000 0x00000000080fffff msi
> > 0x0000000008000000 0x00000000080fffff msi
> > 0x0000000008000000 0x00000000080fffff msi
> > [output trimmed]
>
> But this does not seem correct, you should have a "reserved" region
> covering 0x8000000 as well because you say your platform cannot do DMA
> to 0x8000000 and this is why you are doing all this.
>
> All IOVA that the platform cannot DMA from should be reported in the
> reserved_regions file as "reserved". You must make your platform
> achieve this.
>
Just to double confirm, the expected reserved region should be marked as
"direct" instead of "msi", right?
> > Yes, i tried that,
> >
> > This is how my dts node looked like
> > reserved-memory {
> > faulty_iova: resv_faulty {
> > iommu-addresses = <&pcieX 0x8000000
> > 0x100000>; };
> > ..
> > ..
> > }
> >
> > &pcieX {
> > memory-region = <&faulty_iova>;
> > };
> >
> > I see it working for the devices which are calling
> > iommu_get_resv_regions(), eg if I specify faulty_iova for dma
> > controller dts node then i see an additional entry in the related
> > group
>
> Exactly, it has to flow from the DT into the reserved_regions, that is
> essential.
>
> So what is the problem if you have figured out how to fix up
> /sys/kernel/iommu_groups/Y/reserved_regions?
>
> If you found some cases where you can't get /sys/../reserved_regions
> to report the right things from the DT then that needs to be addressed
> first before you think about fixing SW_MSI.
>
> I very vaguely recall we have some gaps on OF where the DMA-API code
> is understanding parts of the DT that don't get mapped into
> reserved_regions and nobody has cared to fix it because it only
> effects VFIO. You may have landed in the seat that has to fix it :)
>
> But I still don't have a clear sense of what your actual problem is as
> you are show DT that seems reasonable and saying that
> /sys/../reserved_regions is working..
>
> Jason
