On 2026-07-01 at 09:21 +1000, Matthew Brost <[email protected]> wrote... > On Tue, Jun 30, 2026 at 08:39:19AM -0300, Jason Gunthorpe wrote: > > On Tue, Jun 30, 2026 at 07:39:44PM +1000, Dave Airlie wrote: > > > I've been bringing up nouveau on NVIDIA Spark GB10 and fallen down a > > > hole of why is the GPU writing to pages I've released back to the CPU > > > page allocator. I've wasted a lot of time on GPU L2 cache and explicit > > > handling for it, which might be needed but hasn't solved my problem. > > > I've also invalidated and flushed the GPU TLB excessively. > > > > > > Today I finally hit up iommu.strict=1 makes things a lot happier, > > > non-strict IOMMU seems to allow a race between dma_unmap_page and > > > free_page where the unmap goes into the IOMMU flush queue, where the > > > actual unmap are delayed until something triggers a flush later, and > > > the ATS translations stay alive past when they should, and after the > > > page has been allocated by some subsequent user. > > > > It is a driver/device bug if it continues to DMA to memory after > > dma_unmap is called. Strict mode is intended to increase security > > against malicious devices and to aid debugging buggy drivers, it must > > not be relied upon to have any functional effect on a correct driver. > > > > I think I agree here. In Xe, AFAIK, we never call dma_unmap until all > fences have been signaled (dma-fence or preempt-fence mode), after which > GPU translations are rebuilt before the next fence is installed, or > until the GPU page tables have been invalidated (GPU page faults). TTM > is a little harder to reason about, as we DMA-unmap via > ->ttm_tt_unpopulate(), but I just audited the call sites and TTM looks > correct. > > > This is not an API issue or an impedence mismatch, you have to follow > > the DMA API contract when using the DMA API - meaning before unmap is > > called the device is not doing DMA any more. It is a purely a > > driver/device bug that the device is not quieted prior to unmapping. > > > > You cannot fix this with any of your proposed hacks, the driver must > > have control over the device and it must know when the device is done > > DMA. Presumably there is a missing flush or fence that is allowing the > > GPU to continue to DMA even after the OS thinks it has told it to > > stop. There are many unique things about the coherent GPUs, I would > > not be surprised if the driver needs some additional fencing > > operations to clear the fabric beyond simple cache invalidation, but I > > don't know. > > > > Normally strict would convert these bugs from quiet corruptions into > > noisy IOMMU logging, however in this case the GPU is using ATS and the > > IOMMU does not log non-present ATS responses. Instead the GPU's > > internal attempt to DMA will fail and the GPU will do whatever it does > > to propogate that failure. Probably the driver has not noticed the > > fall out, or it ignored it. > > Right. We only have ATS enabled in Xe for experimentation, but whenever > we've had issues with dma_unmap ordering in Xe without ATS, they have > resulted in DMAR-related dmesg spam. > > So TL;DR I'm guessing Nouveau has dma_unmap before GPU idle bug > somewhere.
Agree with most of the above and this seems the most likely explaination. ATS doesn't change much here - most of our driver still goes via dma_map/unmap to access host memory via an IOMMU IOVA so non-present entries there would be fatal at the IOMMU level. So even with ATS enabled a non-present DMA mapping should still result in noisy IOMMU faults. - Alistair > Matt > > > > > Jason
