Hi Robin- > On Jan 26, 2021, at 11:05 AM, Robin Murphy <robin.mur...@arm.com> wrote: > > Implementing .iotlb_sync_map means that a single top-level > iommu_map()/iommu_map_sg() call should still only invoke a single "TLB flush" > (really, any maintenance required for the IOMMU to start using the new > mapping) at the end, regardless of how many internal __iommu_map() calls are > made to satisfy the overall request. If you're seeing something other than > that behaviour (with this series), that implies we've not got things quite > right yet.
The flush is expensive, but it's not the only cost. DMA-mapping a 120KB SGL in a single domain_mapping() call vs. 30 calls is certainly going to be a detectable difference. Naively speaking, if there are more DMA mappings to keep track of because the IOMMU driver isn't coalescing the SGLs the way it did before, that might trigger TLB thrashing on the NIC. > Is there any significant difference between how the NFS read and write paths > make their DMA API calls and/or get their scatterlists in the first place, > that might help shed some light on the curious half-recovery you got? There isn't a difference in the RPC-over-RDMA code. Client-side DMA mapping is handled in net/sunrpc/xprtrdma/frwr_ops.c :: frwr_map() which is used for both I/O directions. On the server, the RDMA core r/w API is used for mapping and then posting RDMA Read and Write work requests. That API appears in drivers/infiniband/core/rw.c , and as far as I understand, the same mapping functions are used for both I/O directions. It's possible that the NIC is doing something different for RDMA Read and RDMA Write, but I don't have much visibility into that. Reads are very different from Writes, which are posted. -- Chuck Lever _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
