On Tue, Jun 30, 2026 at 08:39:19AM -0300, Jason Gunthorpe wrote: > On Tue, Jun 30, 2026 at 07:39:44PM +1000, Dave Airlie wrote: > > I've been bringing up nouveau on NVIDIA Spark GB10 and fallen down a > > hole of why is the GPU writing to pages I've released back to the CPU > > page allocator. I've wasted a lot of time on GPU L2 cache and explicit > > handling for it, which might be needed but hasn't solved my problem. > > I've also invalidated and flushed the GPU TLB excessively. > > > > Today I finally hit up iommu.strict=1 makes things a lot happier, > > non-strict IOMMU seems to allow a race between dma_unmap_page and > > free_page where the unmap goes into the IOMMU flush queue, where the > > actual unmap are delayed until something triggers a flush later, and > > the ATS translations stay alive past when they should, and after the > > page has been allocated by some subsequent user. > > It is a driver/device bug if it continues to DMA to memory after > dma_unmap is called. Strict mode is intended to increase security > against malicious devices and to aid debugging buggy drivers, it must > not be relied upon to have any functional effect on a correct driver. >
I think I agree here. In Xe, AFAIK, we never call dma_unmap until all fences have been signaled (dma-fence or preempt-fence mode), after which GPU translations are rebuilt before the next fence is installed, or until the GPU page tables have been invalidated (GPU page faults). TTM is a little harder to reason about, as we DMA-unmap via ->ttm_tt_unpopulate(), but I just audited the call sites and TTM looks correct. > This is not an API issue or an impedence mismatch, you have to follow > the DMA API contract when using the DMA API - meaning before unmap is > called the device is not doing DMA any more. It is a purely a > driver/device bug that the device is not quieted prior to unmapping. > > You cannot fix this with any of your proposed hacks, the driver must > have control over the device and it must know when the device is done > DMA. Presumably there is a missing flush or fence that is allowing the > GPU to continue to DMA even after the OS thinks it has told it to > stop. There are many unique things about the coherent GPUs, I would > not be surprised if the driver needs some additional fencing > operations to clear the fabric beyond simple cache invalidation, but I > don't know. > > Normally strict would convert these bugs from quiet corruptions into > noisy IOMMU logging, however in this case the GPU is using ATS and the > IOMMU does not log non-present ATS responses. Instead the GPU's > internal attempt to DMA will fail and the GPU will do whatever it does > to propogate that failure. Probably the driver has not noticed the > fall out, or it ignored it. Right. We only have ATS enabled in Xe for experimentation, but whenever we've had issues with dma_unmap ordering in Xe without ATS, they have resulted in DMAR-related dmesg spam. So TL;DR I'm guessing Nouveau has dma_unmap before GPU idle bug somewhere. Matt > > Jason
