On Tue, Aug 03, 2021 at 04:14:57PM +0800, Jason Wang wrote: > > 在 2021/8/3 下午1:51, Chao Gao 写道: > > On Tue, Aug 03, 2021 at 12:43:58PM +0800, Jason Wang wrote: > > > 在 2021/8/3 下午12:29, Chao Gao 写道: > > > > Ping. Could someone help to review this patch? > > > > > > > > Thanks > > > > Chao > > > > > > > > On Wed, Jul 21, 2021 at 03:54:02PM +0800, Chao Gao wrote: > > > > > If guest enables IOMMU_PLATFORM for virtio-net, severe network > > > > > performance drop is observed even if there is no IOMMU. > > > > > > We see such reports internally and we're testing a patch series to disable > > > vhost IOTLB in this case. > > > > > > Will post a patch soon.
[1] > > OK. put me in the CC list. I would like to test with TDX to ensure your > > patch > > fix the performance issue I am facing. > > > Sure. > > > > > > > > > > > > > > > And disabling > > > > > vhost can mitigate the perf issue. Finally, we found the culprit is > > > > > frequent iotlb misses: kernel vhost-net has 2048 entries and each > > > > > entry is 4K (qemu uses 4K for i386 if no IOMMU); vhost-net can cache > > > > > translations for up to 8M (i.e. 4K*2048) IOVAs. If guest uses >8M > > > > > memory for DMA, there are some iotlb misses. > > > > > > > > > > If there is no IOMMU or IOMMU is disabled or IOMMU works in pass-thru > > > > > mode, we can optimistically use large, unaligned iotlb entries instead > > > > > of 4K-aligned entries to reduce iotlb pressure. > > > > > > Instead of introducing new general facilities like unaligned IOTLB entry. > > > I > > > wonder if we optimize the vtd_iommu_translate() to use e.g 1G instead? > > using 1G iotlb entry looks feasible. > > > Want to send a patch? > > > > > > > } else { > > > /* DMAR disabled, passthrough, use 4k-page*/ > > > iotlb.iova = addr & VTD_PAGE_MASK_4K; > > > iotlb.translated_addr = addr & VTD_PAGE_MASK_4K; > > > iotlb.addr_mask = ~VTD_PAGE_MASK_4K; > > > iotlb.perm = IOMMU_RW; > > > success = true; > > > } > > > > > > > > > > > Actually, vhost-net > > > > > in kernel supports unaligned iotlb entry. The alignment requirement is > > > > > imposed by address_space_get_iotlb_entry() and > > > > > flatview_do_translate(). > > > > > > For the passthrough case, is there anyway to detect them and then disable > > > device IOTLB in those case? > > yes. I guess so; qemu knows the presence and status of iommu. Currently, > > in flatview_do_translate(), memory_region_get_iommu() tells whether a memory > > region is behind an iommu. > > > The issues are: > > 1) how to know the passthrough mode is enabled (note that passthrough mode > doesn't mean it doesn't sit behind IOMMU) memory_region_get_iommu() should return NULL if it's passthrough-ed? > 2) can passthrough mode be disabled on the fly? If yes, we need to deal with > them I don't think it happens in reality; e.g. when iommu=pt is set it's set until the next guest reboot. However I don't know whether there's limitation from spec-wise. Also I don't know whether there's special cases, for example when we kexec. I've two questions.. Jason, when you mentioned the "fix" above [1], shouldn't that also fix the same issue, and in a better way? Because ideally I think if we know vhost does not need a translation for either iommu_platform=off, or passthrough, dev-iotlb layer seems an overhead with no real use. The other question is I'm also wondering why we care about iommu_platform=on when there's no vIOMMU at all - it's about why we bother setting the flag at all? Or is it a valid use case? Thanks, -- Peter Xu